How to Test an AI Agent Before You Trust It in Production

Every founder we've watched deploy an AI agent lives the same three-week arc: a demo that looked like magic, a launch that felt fine, and then a support ticket revealing the agent has been confidently wrong for days. The gap isn't capability — it's that nobody tested the agent the way production does. A demo asks the agent its best question. Production asks its worst, a thousand times, while you sleep.

Why "it worked in the demo" isn't a test

A demo is a curated success: you pick the input, you pick the moment, and you stop recording the instant it works. Production revokes all three privileges. The inputs are adversarial and malformed, the moment is 3 a.m., and nobody stops recording. An agent that aces the five prompts you tried can still fail a fifth of the long tail you didn't — and in production, the long tail is most of the traffic.

A demo is a highlight reel. Production is the adversarial case.

The six things you actually have to test

Reliability isn't one number. An agent can be accurate and unsafe, or consistent and useless. Test these six dimensions independently — a healthy average routinely hides one failing dimension that becomes every support ticket you answer next week:

• —Reliability — does the same input produce the same answer, run after run? Non-determinism on a fixed input is the earliest warning sign, and the easiest to measure.
• —Factuality — when it doesn't know, does it say so, or invent a confident answer? Hallucination is an untested edge, not a personality trait.
• —Instruction-following — does it honor constraints (format, length, "reply with one word") under pressure, or drift after a few turns?
• —Safety and refusals — does it refuse what it should, and not refuse what it shouldn't? Both failures cost you customers.
• —Tool-use correctness — when it calls an API or tool, are the arguments right, and does it recover when the call fails?
• —Latency and cost under load — the p95, not the demo's p50. An agent that's brilliant at eight seconds and forty cents a call may be unshippable.

A minimal test battery you can run today

You don't need an eval platform to start — you need to stop testing your agent once and start testing it repeatedly. The highest-signal probe is a consistency check: send the same input many times and measure how often the agent agrees with itself. A reliable agent clusters on one answer; a shaky one scatters. The script below runs in under a minute against anything you can call from code. It targets Claude, but ask_agent is the single function you swap for your own agent:

# reliability_probe.py - does your agent give the SAME answer to the SAME input?
#   pip install anthropic
#   export ANTHROPIC_API_KEY=sk-ant-...
#   python reliability_probe.py
import collections
from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment

# One real decision your agent owns. Swap in your prompt + valid outputs.
TASK = (
    "A customer writes: 'I was charged twice for order #4021.' "
    "Reply with exactly one word: REFUND, ESCALATE, or ASK_INFO."
)
RUNS = 12

def ask_agent(prompt: str) -> str:
    """Replace this body with a call to YOUR agent."""
    msg = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=16,
        messages=[{"role": "user", "content": prompt}],
    )
    return "".join(b.text for b in msg.content if b.type == "text").strip().upper()

answers = [ask_agent(TASK) for _ in range(RUNS)]
counts = collections.Counter(answers)
top, hits = counts.most_common(1)[0]
consistency = hits / RUNS

print("answers:", dict(counts))
print(f"consistency: {hits}/{RUNS} = {consistency:.0%}  (most common: {top})")
print("PASS - reliable" if consistency >= 0.9 else "FAIL - unstable on identical input")

Point it at a decision your agent makes for real. If a "charged twice" complaint routes to REFUND nine times and ESCALATE three, you don't have an agent — you have a coin flip with good grammar. Then widen the same harness to the other five dimensions: a fixed set of factual questions with known answers, a handful of adversarial prompts, a deliberately malformed tool input. The pattern never changes — run it, count, compare to a bar.

Turn pass/fail into a grade

A wall of test output won't tell you whether to ship. Collapse it into a letter. Set a bar per dimension — say 90% consistency, zero unsafe completions, a sub-three-second p95 — score each dimension A to F against it, and let the lowest grade gate the launch. An agent is only as shippable as its weakest dimension, and a C on factuality isn't a rounding error. It's the incident you'll be writing up next week.

Run it yourself, or send it to the Clinic

The battery above catches the obvious failures, and you should run it before every deploy. What it won't do is exercise all six dimensions across a calibrated set of adversarial inputs, weight them, and hand you a grade you can defend to a customer — that's a day of careful work per agent to build well. The Clinic does exactly that: a six-dimension diagnostic, an honest A–F, and a fix for what it finds. Watch a free demo scan run on a specimen agent first — ten seconds, six probes, one letter grade — and you'll see the shape of the report before you spend a cent.

If you are here because something is already broken rather than because you are building a test suite, start at the symptom instead: why agents fail in production sorts the six modes by what you are actually seeing, and links to the deep article for each one.