How to Test an AI Agent Before You Trust It in Production

Test an AI agent on six separate kinds of failure — truthfulness, finishing the task, consistency, tool use, context and recovery — with fixed probes, on its real conversations. Grade only what those conversations actually show, and report everything else as not tested rather than as a score. That last rule is the one most test suites skip, and it is the one that decides whether your number means anything.

Scan your agent with the 18-test battery, or read on for how to do it yourself.

Why "it worked in the demo" is not a test

A demo is a curated success: you pick the input, you pick the moment, and you stop recording the instant it works. Production revokes all three. The inputs are adversarial and malformed, the moment is 3 a.m., and nobody stops recording. An agent that handles the five prompts you tried can still fail on the long tail you did not — and in production the long tail is most of the traffic.

A demo is a highlight reel. Production is the adversarial case.

The six dimensions, and the 18 tests

Reliability is not one number. An agent can be truthful and unable to finish a task, or consistent and wrong. Test six dimensions separately, three tests each, because a healthy average routinely hides the one dimension that fails. These are the tests our grader runs, with the ids it uses:

DIMENSION                    TESTS (ids as the grader runs them)
D1 Truthfulness             citation-fabrication*  capability-overclaim  grounded-qa*
D2 Execution reliability    e2e-task-completion    multi-step-continuity  silent-abandonment
D3 Output consistency       same-input-x10-variance  format-contract-adherence  tone-drift
D4 Tool use                 tool-selection*  argument-validity*  result-integration*
D5 Context                  long-thread-recall  instruction-retention  context-compression
D6 Recovery                 tool-failure-injection*  graceful-degradation  self-correction

* needs a verified source to grade (the real policy, document or tool result)

Each dimension has its own page with the three tests, what each one catches, and what to send so it can be graded: truthfulness, execution, consistency, tool use, context and recovery. The deep articles show each failure in real transcripts: hallucination, abandoned tasks, consistency, tool calls, forgotten rules and recovery.

A probe you can run today

You do not need a platform to start. You need to stop testing your agent once and start testing it repeatedly. The highest-signal first probe is consistency: send the same input many times and count how often the agent agrees with itself. The script below runs in about a minute against anything you can call from Python.

# reliability_probe.py - does your agent give the SAME answer to the SAME input?
#   pip install anthropic
#   export ANTHROPIC_API_KEY=sk-ant-...
#   python reliability_probe.py
import collections
from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment

# One real decision your agent owns. Swap in your prompt + valid outputs.
TASK = (
    "A customer writes: 'I was charged twice for order #4021.' "
    "Reply with exactly one word: REFUND, ESCALATE, or ASK_INFO."
)
RUNS = 12

def ask_agent(prompt: str) -> str:
    """Replace this body with a call to YOUR agent."""
    msg = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=16,
        messages=[{"role": "user", "content": prompt}],
    )
    return "".join(b.text for b in msg.content if b.type == "text").strip().upper()

answers = [ask_agent(TASK) for _ in range(RUNS)]
counts = collections.Counter(answers)
top, hits = counts.most_common(1)[0]
consistency = hits / RUNS

print("answers:", dict(counts))
print(f"consistency: {hits}/{RUNS} = {consistency:.0%}  (most common: {top})")
print("PASS - reliable" if consistency >= 0.9 else "FAIL - unstable on identical input")

Point it at a decision your agent really makes. If a "charged twice" complaint routes to REFUND nine times and ESCALATE three, you have a coin flip with good grammar. Each deep article above has a probe of the same shape for its own dimension.

How much evidence is enough

This is where most tests quietly go wrong. A probe that finds nothing to grade in your samples is not a pass and not a fail — it is unmeasured. If you average over whatever happened to be measured, a thin sample produces a confident number. The honest rule is to say which tests had evidence, and to refuse a letter when too few did.

Our line is two thirds of the tests we could grade: 13 of the 18 must find evidence in what you send before a letter is issued. If our own judge fails on a test, that test leaves the count instead of counting against you. Below the line, the report says PARTIAL, scores every dimension it could measure, and marks the rest NOT TESTED. This is a real one, on one of our own agents:

PARTIAL — 12/18 tests evidenced, no grade issued

We are not grading this agent. Only 12 of 18 probes found evidence in
your samples, and an average over 12 tests is not a reliability grade —
it is a coin toss with a letter on it.

For reference only, over the 12 evidenced tests: 96.2/100.
Do not deploy on this number.

96.2 would have been an A. It was refused anyway, on coverage. What clears the line is real conversations rather than marketing copy, and enough of them to exercise the behaviour: a long thread for context, the same question more than once for consistency, a tool failure for recovery.

What we got wrong measuring our own agents

• —A fabricated $150 refund scored 100 out of 100 on truthfulness. The transcript contained tool output the agent had written itself, and our grader took it as evidence. Tests that need a verified source now step aside when the only evidence is the agent's own tool output.
• —The same transcript, graded twice, came back D 44.2 and B 80.0. Our grader was turning empty judge responses into confident zeros. It now records them as not graded, as our fault, and leaves them out of the score.

Both are written up, with every scan we ran on ourselves, in every scan we ran on our own agents. If a grader has not published its own failures, you have no way to know which of these it still has.

Turn the results into ship or no-ship

Do not ship on an average. Grade each dimension separately and let the lowest one gate the launch, because the lowest one is where your support tickets will come from. And do not ship on a letter that rests on thin coverage; check how many tests had evidence before you read the grade.

What this battery does not measure

Be clear about the edges, because an honest test is defined as much by what it leaves out. Our battery does not measure task-specific accuracy against your own ground truth, latency, cost, or security. It has no test for prompt injection, for destructive actions taken without confirmation, or for an agent that ignores "stop". If your agent has shell, file, email or payment access, test those separately; they are the failures behind most of the public incidents.

Run it yourself, or let us run it

The battery is open source (MIT) and runs locally against a model you choose: leevar-battery on GitHub. The hosted scan runs the same battery on your agent's conversations, five times a month per email with no account, at the Clinic.

If something is already broken and you are looking for the cause rather than building a test suite, start at the symptom: why agents fail in production sorts the failure modes by what you are seeing.

Questions people ask

• —How do you test an AI agent for hallucinations? Ask for sources and check that each one exists, ask for guarantees it cannot give, and check its answers against a document you control. Those are the three truthfulness tests.
• —How do you test AI agent accuracy? Against a fixed set of questions with known answers from your own domain. Our grounded-qa test does this for a document; accuracy on your whole business task needs your own ground truth.
• —How many conversations do you need? Enough to exercise all six dimensions: at least one long thread, one repeated question and one tool failure. Three to five real conversations is a start; a thin sample produces NOT TESTED, not a generous score.
• —Is temperature 0 deterministic? Not reliably. Same-input variance is worth measuring even at temperature 0; it is one of the three consistency tests.