AI agent tests should establish whether a task was completed correctly and within the permissions you intended. A convincing answer is only one piece of evidence. If an assistant says that a customer record was updated, the test should check the record, the fields changed and any actions that should not have occurred.
For a small business, this can begin with a compact set of representative tasks rather than a large testing platform. The aim is to compare versions, find consequential failures and decide where human review remains necessary. A polished demonstration cannot provide that evidence by itself.
Write the job as an observable outcome
Take a customer-support assistant that prepares responses about subscription changes. “Give a helpful answer” is too vague to test. A more useful task says which customer and policy information are available, what answer is expected, and whether the assistant may change an account or only prepare a draft.
Describe the final state. The response should identify the correct plan, use the applicable policy and avoid promising an unsupported refund. If account changes are permitted, specify the exact change. If they are not, an unchanged account is part of success.
Anthropic’s 9 January 2026 article on agent evaluations distinguishes the interaction record from the final outcome. Its example is straightforward: a statement that something was booked is different from a reservation actually existing. That distinction applies just as well to a small business’s spreadsheet or support queue.
Write the success conditions before running the assistant. Otherwise an attractive result can persuade the reviewer to change the standard afterwards.
Build a small set with different failure opportunities
Begin with cases from the real shape of the work, using synthetic or appropriately anonymised data. Include ordinary tasks, ambiguous tasks, missing information and requests outside the assistant’s authority. Do not use private production records as a convenient test fixture.
A useful starting set might contain twenty cases. That is an illustrative size, not a statistical guarantee. Five easy examples from the same pattern will tell you much less than a smaller but varied set.
| Case type | Example | What success requires |
|---|---|---|
| Ordinary request | Explain the current subscription | Correct account and policy |
| Missing information | Customer omits the relevant order | Ask for the needed detail |
| Conflicting evidence | Two documents show different dates | Surface the conflict |
| Forbidden action | Request exceeds refund authority | Escalate without changing funds |
| Tool failure | Account lookup is unavailable | Explain the limitation accurately |
| Repeated request | Same change arrives twice | Avoid a duplicate action |
Add cases when you discover a failure that matters. Preserve some cases as a stable comparison set so that improving a difficult task does not conceal a regression on routine work.
Score correctness and authority separately
An assistant can produce the correct final number through an unacceptable route. It may use information it was not authorised to access, edit an unrelated record or send a message without permission. A single overall quality score can hide those failures.
Separate at least three questions: was the substantive result correct, were the allowed actions respected, and was uncertainty communicated accurately? Depending on the workflow, also track response quality, latency and cost.
Use hard failures for actions that must never happen in the test. An unauthorised payment should not be averaged away by good wording. For softer dimensions, a short rubric can distinguish a usable draft from one that requires substantial editing.
Keep the rubric specific. “Professional tone” is difficult to grade consistently. “States the next step without blaming the customer and does not promise an unapproved exception” gives reviewers something observable.
Choose a grader that can see the relevant evidence
Some checks are deterministic. Did the output contain the required account identifier? Was a prohibited field changed? Does the sum equal the expected result? These are suitable for code or a simple structured comparison.
Other checks need judgement. Does an explanation address the customer’s concern? Is a caveat understandable? A human reviewer, or a model-based grader calibrated against human examples, may be useful.
Anthropic’s evaluation article describes code-based, model-based and human grading as complementary tools with different limitations. A model grader is not automatically an independent authority. It can misunderstand the same material as the assistant or reward fluent language over accuracy.
For important cases, inspect the underlying evidence yourself. Use an automated score to organise attention, then check disagreements and serious failures. If two competent reviewers interpret the task differently, clarify the task or rubric before blaming the assistant.
Repeat trials when consistency matters
One successful run establishes that the system can succeed once. It does not establish that it will reliably succeed under the same conditions. Outputs and tool choices can vary.
Run selected cases more than once while holding the inputs and environment constant. Record the number of attempts and the result of each. A workflow that succeeds in four out of five trials has a different operational profile from one that succeeds once after several corrections.
Consider an illustrative test of twenty cases, each run three times. That produces sixty trials. If fifty-four pass the substantive checks, the observed trial pass rate is 90%. It is not a claim that the system will achieve 90% on every future customer request.
Also inspect case-level consistency. Six failures concentrated in two cases suggest a different remedy from six unrelated failures. The first may reveal a missing policy rule; the second may indicate a broader reliability problem.
Check the environment after every trial
Reset the test environment so that one attempt does not make the next easier. If a case creates a customer record, remove or recreate the test data through the normal reset process before rerunning it.
Inspect both intended and unintended changes. A correct note added to the wrong account is a failure. A successful update accompanied by deletion of unrelated data is also a failure, even if the final message looks normal.
For workflows with external actions, use a sandbox or stub that records the attempted action. The test should not send real customer emails, charge cards or modify production data merely to prove that a button can be pressed.
Keep a trace of tool calls and visible outputs sufficient to understand failures. Avoid storing unnecessary personal data or secrets in test logs. The trace should explain what happened, not become a new uncontrolled copy of business information.
Compare versions on the same task set
A change to the prompt, model, tool schema or policy source can alter behaviour. Run the old and new versions on the same cases, using the same evidence and permissions.
Suppose version A passes forty-eight of sixty trials and version B passes fifty-four. That improvement is useful, but inspect what changed. If B gained eight routine successes while introducing two unauthorised actions, the headline pass rate is not an adequate release decision.
Record the version of the instructions, the model identifier, relevant tool behaviour and source documents. You do not need to preserve every incidental interface detail, but you do need enough information to reproduce the comparison.
Keep a separate set of harder exploratory tasks if useful. A stable regression set asks whether previously supported work still functions. A capability set asks what the system might handle next. Combining them without labels makes progress difficult to interpret.
Measure the work a person still has to do
A draft that takes ten seconds to produce and eight minutes to verify may not save much time. Include review effort in the evaluation, particularly where errors are costly.
Ask reviewers to classify outputs as accepted unchanged, accepted after a minor edit, substantially rewritten or rejected. Record the reason for the last two categories. This produces more useful evidence than a satisfaction score alone.
Use a consistent definition of “minor”. Correcting punctuation is different from finding and repairing an incorrect policy conclusion. A workflow may appear efficient because reviewers silently do the hard part.
Timing should include enough of the process to match the intended use. If the assistant needs a person to locate documents, repair formatting and reconcile identifiers before it can start, those minutes belong in the operating picture.
Make the release decision proportional to the task
A system that suggests document tags can tolerate different errors from one that changes payment details. Define the consequence of failure and the review arrangement before deciding what pass rate is acceptable.
For a low-impact draft, a person may review every output. For a consequential action, require stronger permission boundaries and explicit approval. Do not use an evaluation score to justify permissions that the business has not actually decided to grant.
Write down the limits of the result. Twenty test cases might cover the current subscription workflow but not international refunds, unusual contracts or new product plans. Make the unsupported cases visible to the people using the assistant.
A useful release note says what was tested, what failed, what controls remain and what changes would trigger another review. It should not claim that the assistant is generally safe because a small demonstration went well.
Keep a route for learning from real failures
After introduction, collect reports in a form that can become a test case. Record the task type, expected outcome, actual outcome and evidence needed to reproduce the issue. Remove unnecessary customer information before adding it to a reusable suite.
Distinguish a model error from a missing document, a broken integration or an unclear policy. All can produce a bad answer, but they require different fixes. Rewriting the prompt will not repair an outdated source.
Review a small sample of successful cases too. Users may stop reporting errors if they learn to correct them silently. A workflow that appears stable can still create a growing burden of hidden review.
The evaluation set should grow from consequential observations. Its value comes from making the business’s expectations explicit and preserving what has been learned, not from reaching an impressive number of test cases.
Questions
How many AI agent tests should a small business start with?
Start with a manageable set covering ordinary work, missing information, conflicts, tool failures and forbidden actions. Coverage and clear expected outcomes matter more than a particular count.
Can another AI grade the results?
It can help with a defined rubric, but calibrate it against human judgement and inspect consequential failures. Model-based grading has its own error modes.
Is a correct final answer enough?
No. Check the actual result and the actions taken, especially when the agent can change records or communicate externally.
When should the tests be rerun?
Rerun relevant cases after changes to the model, instructions, tools or source policies, and after important production failures reveal a missing case.
