Agents drift. A prompt tweak that improves one case can quietly break three others. Tests catch that before the regression ships. Apragya AI's test harness lets you author golden test cases that run on every Build, plus larger eval suites that run on a schedule.
Golden tests
- Each test is an (input, expected) pair. The runner fires the agent against the input and decides pass/fail by one of three methods:
- Exact match - the agent's output must equal a reference string (rare; useful for templated outputs)
- Semantic match - an LLM judge compares output to expected for semantic equivalence
- Assertion functions - JavaScript that inspects the run output + trace and returns true/false (most flexible; preferred)
Authoring a golden test
Agent Builder > Tests tab > New Test. Name it after the scenario (e.g., "under-threshold invoice posts to ERP"). Drop in the input payload, pick the pass criterion, save. The Builder walks you through the assertion shape so you don't need code to describe what "success" looks like.
Eval runs
Beyond per-build smoke tests, schedule nightly eval runs that exercise the agent against a much larger test set (50-200 cases) and surface aggregate quality metrics - pass rate, cost variance, latency distribution. Configure on the Tests tab > Schedule Eval Run.
Reading the results
The Tests tab shows pass rate over time + a per-version diff - which build regressed which test. Click into a failed test to see the actual run trace, the assertion that failed, and the diff vs the last passing run.
Blocking the bad publish
Failed tests block Publish by default. Override via the Force Publish button - the override is recorded in the AgentVersion audit trail with the actor and timestamp, so the regression is at least traceable if it makes it to production.