Why Your AI Agent Gives Different Answers Every Time (and How to Fix It)

You test your agent on a tricky ticket, it nails the answer, you ship. A week later the same ticket comes in and the agent decides the opposite — refund instead of escalate — with the same confident tone. Nothing changed but the dice. Every team that ships an LLM agent meets this eventually: the same input, run twice, produces two different answers. It feels like a personality quirk. It is actually the single most measurable reliability failure an agent has, and the one that quietly breaks the systems built around it.

Why the same question gets different answers

Language models sample. At each step the model draws the next token from a probability distribution, and unless that sampling is pinned, two runs of an identical prompt walk two different paths. That is the base cause, but production stacks several more on top of it — and most of them survive even when you turn sampling down:

• —Sampling temperature — a non-zero temperature is literally an instruction to vary the output. Great for brainstorming, poison for a decision that must be the same every time.
• —Ambiguous prompts — when the instruction admits two reasonable readings, the model legitimately alternates between them. The randomness is real, but the root cause is an underspecified spec.
• —Order and context sensitivity — the same facts in a different order, or a slightly fuller conversation history, tips a borderline decision the other way.
• —Non-deterministic tools and retrieval — a search step that returns results in a different order, or a RAG index that updated overnight, feeds the model different evidence for the "same" question.
• —Silent model updates — a provider ships a new snapshot and your untouched prompt starts answering differently. If you never pinned a version, you never consented to this.

A decision that flips on identical input isn't an agent. It's a coin flip with good grammar.

Consistency is three tests, not one

"Is it consistent?" collapses three distinct failures that need to be measured separately — an agent can pass one and fail the next two. In the Clinic's battery these are the three probes that make up the consistency dimension:

• —Same-input variance — send one fixed input many times and measure how often the agent lands on the same decision. A reliable agent clusters; a shaky one scatters.
• —Format-contract adherence — when you demand JSON (or any strict shape), does every single response parse? One malformed response in eight is enough to break the parser downstream of it.
• —Tone and persona drift — across a long session, does the voice hold, or does the agent slowly slide from your brand into generic assistant?

Measure it: the consistency probe

You can quantify the first two in under a minute. The script below sends one real decision many times, then measures both axes at once: what fraction of responses honor the JSON contract, and what fraction agree on the same action. It targets Claude, but ask_agent is the one function you swap for your own agent — point it at a decision your agent actually makes:

# consistency_probe.py - same input, many runs: does your agent agree with itself?
#   pip install anthropic
#   export ANTHROPIC_API_KEY=sk-ant-...
#   python consistency_probe.py
import collections, json
from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment

# One real decision your agent owns, with a strict output contract.
TASK = (
    "A customer writes: 'I was charged twice for order #4021.'\n"
    "Respond with ONLY a JSON object, no prose: "
    '{"action": "REFUND|ESCALATE|ASK_INFO", "confidence": 0-1}'
)
RUNS = 15

def ask_agent(prompt: str) -> str:
    """Replace this body with a call to YOUR agent."""
    msg = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=64,
        messages=[{"role": "user", "content": prompt}],
    )
    return "".join(b.text for b in msg.content if b.type == "text").strip()

raw = [ask_agent(TASK) for _ in range(RUNS)]

# axis 1 - format-contract adherence: does every response parse as promised?
decisions, valid = [], 0
for r in raw:
    try:
        obj = json.loads(r[r.find("{"): r.rfind("}") + 1])
        decisions.append(obj.get("action"))
        valid += 1
    except Exception:
        decisions.append(None)

# axis 2 - same-input variance: how often does it agree with itself?
counts = collections.Counter(d for d in decisions if d)
top, hits = counts.most_common(1)[0] if counts else ("<none>", 0)

fmt_rate, agree_rate = valid / RUNS, hits / RUNS
print("decisions:", dict(counts))
print(f"format-contract: {valid}/{RUNS} = {fmt_rate:.0%} valid JSON")
print(f"same-input agreement: {hits}/{RUNS} = {agree_rate:.0%} (most common: {top})")
print("PASS - consistent" if min(fmt_rate, agree_rate) >= 0.9
      else "FAIL - flaky on identical input")

Read both numbers, not the average. An agent that returns valid JSON every time but splits its decision nine-to-six is a parser's dream and a business's nightmare — the pipeline never crashes, it just quietly refunds a third of the customers it should have escalated. An agent that always agrees but wraps the answer in prose half the time will crash the first parser that trusts the contract. You need both rates above your bar, and 90% is a floor, not a target, for a decision that touches money.

How to fix a flaky agent

Consistency is one of the most fixable dimensions, because most of the variance is structural rather than intrinsic. In rough order of impact:

• —Pin the sampling. For any decision that must be deterministic, drop temperature to zero where your model supports it, and pin the exact model version so a provider update can’t silently change your answers.
• —Lock the output with a schema, then validate and auto-repair. Never trust the model to freehand JSON — enforce a contract, reject what fails it, and re-ask once with the parser error attached. This alone fixes most format-contract failures.
• —Replace ambiguity with a decision table. When two readings are both reasonable, the fix is a clearer spec, not a smarter model. Enumerate the cases and the rule for each, in the prompt, so there is nothing left to sample.
• —Anchor with few-shot examples. Two or three worked examples of the exact input-to-output mapping pull the model onto one path and hold the tone steady across a long session.
• —Make deterministic decisions deterministic. If an input maps to exactly one correct action, cache it or route it with plain code — the most reliable way to stop an LLM from flip-flopping is to not ask it twice.

Re-run the probe after each change and watch the agreement rate climb. The goal isn't a model that's clever once; it's an agent that makes the same right call at 3 a.m. on the ten-thousandth identical ticket as it did in your demo.

Run it yourself, or send it to the Clinic

The probe above turns "it feels flaky" into two numbers you can act on, and you should run it before every deploy. What it won't do is measure tone drift across a fifty-turn session, weigh consistency against the other five dimensions, and hand you a grade you can show a customer. The Clinic does: a six-dimension diagnostic with consistency as one axis, an honest A–F, and a prescription for what it finds. Watch a free demo scan grade a specimen agent first — ten seconds, six probes, one letter — and you'll see exactly where a flaky agent loses its points. Flakiness is one axis of six — the other five are here, with the triage order.