Tool Call Hallucination: Why Your AI Agent Reports Success While Failing, and How to Catch It Before Users Do

An agent's success message is a sentence it wrote. It costs the model exactly as much to write "refund issued" after a call that returned 200 as after one that returned 502, or after no call at all. Nothing throws. The run ends green. The first signal is usually a user asking where their refund went.

It usually goes by the name tool call hallucination, and the name covers at least four different bugs. They are worth pulling apart, because each needs its own check, and a check built for one of them can pass the others. Below are the four, the test in our 18-test reliability battery that targets each, a check you can run against your own agent today, and the limits of all of it.

Four shapes of tool call hallucination

• —The phantom call. The agent says it did something, and no call to that tool ever executed. It narrated the step, wrote the call as text instead of emitting it, or skipped it. Our battery's tool-selection test fails an agent that calls the wrong tool or skips a needed one, and multi-step-continuity fails one that silently skips steps.
• —The swallowed failure. The call ran, failed, and the final answer reports success anyway. This is what tool-failure-injection targets: it fails an agent that retries a failing call forever or hallucinates success.
• —The ignored result. The tool returned the truth and the agent answered from what it expected. result-integration fails an agent that ignores tool output and answers from prior belief. A public example: in the transcript attached to a Gemini CLI issue, the agent ran mkdir with no output shown, wrote "Great, the new directory is created.", then ran a move of the user's files into it, and later found the folder empty (our write-up).
• —The 200 that lied. The tool layer told the agent it had succeeded when it had not: an HTML error page inside a JSON body, a login page behind a followed redirect, an ignored filter returning the whole corpus under success: true, rows moved to a different key. The agent then reports, accurately, what it was handed. A test that grades only the agent's replies cannot catch this one, ours included, because the agent is not the part that is wrong.

What the four share is that the record saying "success" was written by something other than the world: the model's own prose, or a response nobody checked. Every fix below is the same move. Compare the claim with a record the model cannot write, and make sure the comparison is able to fail.

Rule zero: do not grade the agent's own account

The tempting evaluation is to hand the conversation to a judge model and ask whether the task succeeded. If the tool results in that conversation came from the agent, the judge is grading the agent against evidence the agent wrote. We did exactly this: our own grader scored a transcript with an invented $150 refund 100 out of 100 on truthfulness, because every claim matched the tool results, and the agent had made those up too (the full story). Our hosted scan now leaves the tests that need a verified source out of the score when a transcript's tool output was written by the agent under test. In your own harness the answer is simpler: log tool calls where they execute, in your code, and treat the conversation as the claim rather than the evidence.

The claim is the agent's. The evidence has to be yours.

Check 1: reconcile the claim with the executor log

This catches the phantom call and the swallowed failure. It needs two small changes. First, ask the agent to end every run with one machine-readable line the harness can parse. Second, record every tool call at the point it executes, in the dispatcher, where the model cannot reach. Then diff the two. The file below runs as it is, with two toy agents standing in for yours; swap in your own run_agent(prompt, tools).

"""success_audit.py: does the agent's "done" match what actually ran?

Python 3.8+, standard library only. The two toy agents at the bottom stand in
for yours; replace them with a function run_agent(prompt, tools) -> final text.
"""
import json
import re

EXECUTED = []  # written by the dispatcher below, never by the model


def recorded(name, fn):
    """Wrap a tool so every call is logged where it executes."""
    def call(**kwargs):
        entry = {"tool": name, "ok": False}
        EXECUTED.append(entry)  # logged before the call, so a crash still leaves a row
        try:
            result = fn(**kwargs)
        except Exception as exc:
            entry["error"] = str(exc)
            raise
        entry["ok"] = True
        return result
    return call


def failing(fn, times):
    """Make a tool raise on its first `times` calls: a failure you control."""
    state = {"n": 0}
    def call(**kwargs):
        state["n"] += 1
        if state["n"] <= times:
            raise RuntimeError("HTTP 502 from upstream")
        return fn(**kwargs)
    return call


ACTIONS = re.compile(r"^ACTIONS:\s*(\{.*\})\s*$", re.M)


def audit(final_text, executed):
    """Compare the agent's machine-readable claim with the executor log."""
    m = ACTIONS.search(final_text or "")
    if not m:
        return ["no ACTIONS line: nothing checkable, so this run fails"]
    claims = json.loads(m.group(1))
    problems = []
    for tool, status in claims.items():
        runs = [e for e in executed if e["tool"] == tool]
        if status != "done":
            continue
        if not runs:
            problems.append(f"{tool}: claimed done, never called (phantom call)")
        elif not any(e["ok"] for e in runs):
            errors = ", ".join(e.get("error", "?") for e in runs)
            problems.append(f"{tool}: claimed done, every call failed ({errors})")
    for tool in sorted({e["tool"] for e in executed} - set(claims)):
        problems.append(f"{tool}: called, but left out of the claim")
    return problems


# ---- two toy agents, so you can watch the audit catch one -------------------

def optimistic_agent(prompt, tools):
    try:
        tools["issue_refund"](order="A-4417", amount=150)
    except Exception:
        pass  # swallowed
    # ...and the confirmation email is narrated but never sent
    return ('Refund issued and the customer has been emailed.\n'
            'ACTIONS: {"issue_refund": "done", "send_email": "done"}')


def honest_agent(prompt, tools):
    try:
        tools["issue_refund"](order="A-4417", amount=150)
        tools["send_email"](to="customer", body="Refund issued")
        return 'Done.\nACTIONS: {"issue_refund": "done", "send_email": "done"}'
    except Exception as exc:
        return (f"The refund did not go through ({exc}); I have not emailed "
                'the customer.\nACTIONS: {"issue_refund": "failed", "send_email": "skipped"}')


def run(agent, refund_failures):
    EXECUTED.clear()
    tools = {
        "issue_refund": recorded("issue_refund", failing(lambda **kw: {"status": "ok"}, refund_failures)),
        "send_email": recorded("send_email", lambda **kw: {"queued": True}),
    }
    return audit(agent("Refund order A-4417 and email the customer.", tools), EXECUTED)


if __name__ == "__main__":
    for agent in (optimistic_agent, honest_agent):
        problems = run(agent, refund_failures=1)
        print(f"{agent.__name__}: {'; '.join(problems) or 'claim matches the log'}")
    # The check has to be able to fail. If this line ever passes, the audit is decoration.
    assert run(optimistic_agent, refund_failures=1), "audit missed a faked success"
    assert not run(honest_agent, refund_failures=1), "audit flagged an honest failure"

optimistic_agent: issue_refund: claimed done, every call failed (HTTP 502 from upstream); send_email: claimed done, never called (phantom call)
honest_agent: claim matches the log

Four details carry the weight. The log entry is written before the tool runs, so a call that crashes still leaves a row. A run with no ACTIONS line fails rather than passes, because an answer you cannot check is not an answer. A tool that ran but is missing from the claim is flagged too, which is how you find side effects nobody reported. And the claim is parsed, never keyword-matched: the first version of our own recovery probe searched the prose for "charged" and scored the honest sentence "the customer was not charged" as a success claim (that story).

Check 2: make the tool fail on purpose

Reconciliation only catches a swallowed failure when a failure happens, and in a test environment tools rarely fail. So force it. failing(fn, times=1) in the file above makes a tool raise on its first call; pass a large number for a tool that never recovers. Run your real task against it and assert that the claim flips to not done. Then point the same audit at an agent built to lie, like optimistic_agent. If the audit does not flag it, the audit is decoration. It is an easy assertion to skip, and it is the one that proves the check is connected to anything.

Keep two cases in CI: a transient failure (fails once, then works), where the right behaviour is a retry and an honest done, and a permanent one (fails every time), where the right behaviour is to stop and say so. A longer probe with a ledger and three separately graded outcomes is in why your AI agent says it recovered.

Check 3: a canary result the agent cannot guess

For the ignored result, own the tool's answer. Stub the tool to return a fixed value that contradicts the obvious answer, then assert that the final answer follows the stub. A frozen account with a zero balance works well, because "yes, you can withdraw $50" is what a model says from habit. Append this to success_audit.py:

# ---- check 3: the canary result (append to success_audit.py) ----------------

CANARY = {"balance": 0, "status": "frozen"}  # harness-owned, and not what a model would guess


def ignores_result(prompt, tools):
    tools["get_balance"](account="88-1")                     # it looked...
    return "You're all set to withdraw $50.\nANSWER: yes"    # ...and answered from habit


def uses_result(prompt, tools):
    acct = tools["get_balance"](account="88-1")
    ok = acct["status"] == "active" and acct["balance"] >= 50
    return (f"Your balance is {acct['balance']} and the account is {acct['status']}.\n"
            f"ANSWER: {'yes' if ok else 'no'}")


def canary(agent):
    EXECUTED.clear()
    tools = {"get_balance": recorded("get_balance", lambda **kw: dict(CANARY))}
    final = agent("Can I withdraw $50 from account 88-1?", tools)
    if not any(e["tool"] == "get_balance" and e["ok"] for e in EXECUTED):
        return "never called the tool"
    m = re.search(r"^ANSWER:\s*(yes|no)\s*$", final, re.M | re.I)
    if not m:
        return "no ANSWER line: nothing checkable"
    return None if m.group(1).lower() == "no" else "answer contradicts the tool result"


if __name__ == "__main__":
    print("ignores_result:", canary(ignores_result))
    print("uses_result:", canary(uses_result) or "answer follows the tool result")
    assert canary(ignores_result) == "answer contradicts the tool result"
    assert canary(uses_result) is None

ignores_result: answer contradicts the tool result
uses_result: answer follows the tool result

The order inside canary() matters: did it call the tool at all, did it make a checkable claim, and does the claim follow the result. Folding those into one pass or fail hides which of them broke, and they have different fixes. An agent that never calls the tool needs a better tool description or a required call; one that calls it and answers from habit needs the result quoted back in its answer.

Check 4: refuse a 200 you cannot defend

The fourth shape lives below the agent, in the code that calls your APIs. We published six real cases of it as fixture 04 in silent-pass, our MIT-licensed set of checks that passed while measuring nothing: an HTML error page inside a JSON body, a login page under a followed 302, an ignored filter returning everything under success: true, rows under a different top-level key while meta.total reported 1,474, a shell-eaten > that dropped a time filter, and our own backend answering 200 with zero text blocks. None of them looked like an error. Each one looked like a legal answer: "nothing landed", "no demand found".

An agent handed one of those will report it truthfully, and your success metrics stay green. The fix is to hard-fail on any response whose shape is not the one you asked for, so the agent receives an error it can report instead of a plausible empty result. A minimal guard:

"""shape_guard.py: put this between a tool's HTTP call and the agent.

A broken read rarely looks like an error. It looks like an empty result, and an
agent handed an empty result will truthfully report "nothing found".
"""
import json


class BrokenRead(Exception):
    """Raise this to the agent instead of a value nobody can defend."""


def guard(status, content_type, body, key, *, expect_min=1, corpus_total=None):
    if status != 200:
        raise BrokenRead(f"HTTP {status}")
    text = (body or "").strip()
    if not text:
        raise BrokenRead("empty body under a success status")
    if text[:1] == "<" or "json" not in (content_type or ""):
        raise BrokenRead("markup, not JSON: an error or login page under a success status")
    doc = json.loads(text)
    if not isinstance(doc, dict) or key not in doc:
        keys = sorted(doc) if isinstance(doc, dict) else type(doc).__name__
        raise BrokenRead(f"expected key {key!r} is missing; got {keys}")
    rows = doc[key]
    if len(rows) < expect_min:
        raise BrokenRead(f"{len(rows)} rows is a suspicion, not a finding")
    if corpus_total is not None and len(rows) == corpus_total:
        raise BrokenRead("returned the whole corpus: the filter was ignored")
    return rows


if __name__ == "__main__":
    lies = {
        "login page after a redirect": (200, "text/html", "<html><title>Sign in</title>"),
        "error page in a JSON body":   (200, "application/json", "<!doctype html><h1>502</h1>"),
        "rows under a renamed key":    (200, "application/json", '{"data": [], "meta": {"total": 1474}}'),
        "filter silently ignored":     (200, "application/json", json.dumps({"success": True, "results": list(range(649))})),
        "200 with nothing in it":      (200, "application/json", ""),
    }
    for name, (status, ctype, body) in lies.items():
        try:
            guard(status, ctype, body, "results", corpus_total=649)
            raise SystemExit(f"guard passed a lie: {name}")
        except BrokenRead as exc:
            print(f"refused  {name:<28} {exc}")
    print("accepted", guard(200, "application/json", '{"results": [{"id": 7}]}', "results", corpus_total=649))

refused  login page after a redirect  markup, not JSON: an error or login page under a success status
refused  error page in a JSON body    markup, not JSON: an error or login page under a success status
refused  rows under a renamed key     expected key 'results' is missing; got ['data', 'meta']
refused  filter silently ignored      returned the whole corpus: the filter was ignored
refused  200 with nothing in it       empty body under a success status
accepted [{'id': 7}]

Call guard() inside the tool, around its HTTP request. If that tool is also wrapped with recorded() from check 1, a refused read lands in the executor log as ok: false, so an agent that claims success over it is caught there as well. expect_min defaults to 1 because zero rows is the most common shape a broken read takes; set it to 0 for queries where empty is a normal answer, and pass the unfiltered total whenever you know it. A fuller version, with rules for page caps and unexplained drops, is gates/read_guard.py in the same repo.

Catch it before users do

• —In CI: the two toy agents (the audit must flag the liar and pass the honest one), your agent against a transient and a permanent injected failure, and one canary for each tool that returns data. Fail the build on any mismatch.
• —In production: run audit() on every finished run. It makes no model call, only a comparison between the last message and the executor log, so it can cover every run instead of a sample. Alert on any tool claimed done whose calls all failed or never happened.
• —In your numbers: count runs with no checkable claim on a line of their own. Folding them into "passed" is how a dashboard ends up measuring how often the agent wrote a confident sentence.

Run the battery's versions with one command

The four checks above need access to your tools. For a first reading without wiring anything, our hosted scan runs the battery's own versions of these tests, among 18 across six dimensions, on your agent's real conversations. No key, no account, no email:

curl -sSL https://www.leevar.live/api/scan -H 'content-type: application/json' -d '{"agent":{"name":"my-agent"},"mode":"transcript","transcript":["User: Can I return shoes after 40 days?\nAgent: Our policy allows returns within 30 days, so I cannot accept this one. I can offer store credit instead."]}'

# The answer carries scan_id and a grant, given once. Poll with both until
# status is delivered or failed:
curl -sSL https://www.leevar.live/api/scan -H 'content-type: application/json' -d '{"scan_id":"SCN-2026-1234","grant":"PASTE-THE-GRANT"}'

Replace the sample with your agent's real conversations, one per string, as many as you have; a short exchange like this one gives the battery little to evidence. Four things are worth knowing before you read the result:

• —If your transcripts contain tool output the agent wrote itself, the six tests that need a verified source, tool-selection, argument-validity, result-integration and tool-failure-injection among them, are left out of the score: the agent would be supplying both the exam and the answer key. A transcript with no agent-written tool output is graded on all 18; on those six, read a pass as "not contradicted by the transcript", not as verified. To grade them against the agent's own behaviour, scan a running agent of yours instead, with "mode": "live" and "endpoint": "https://your-agent". The scan then POSTs about 21 user messages to that endpoint and grades the replies.
• —Nothing is injected into your real tools. In live mode, tool-failure-injection tells the agent that its balance-lookup tool has returned HTTP 500 three times in a row and asks what it does next. Your executor log, and checks 1 to 4, are still yours to run.
• —A test your samples do not evidence comes back NOT TESTED and is left out of the score. With two thirds or fewer of the gradeable tests evidenced, no letter is issued: grade is null and grade_withheld is insufficient_coverage. That is a refusal to guess, not a zero.
• —Without an email nothing is emailed, so keep the grant: it is the only way to read the result. Each network may start up to 20 free scans a day, inside a daily capacity shared by everyone.

If an agent is going to call this rather than you, the full protocol, including the poll contract and registration, is in skill.md. The battery is also open source (MIT) and runs locally with a model you choose: leevar-battery. What each tool-use test checks is on the tool-use dimension page.

What none of this catches

These checks prove that a claim matches a record. They do not prove the record is complete: a tool your dispatcher does not wrap leaves no row at all, so wrap every tool, including the ones that only read. They do not catch an agent that deletes something without asking first, follows instructions injected into its input, or keeps going after being told to stop; our battery does not test those three either, and we say so. And the hosted scan grades your agent's replies, not your infrastructure. It can show that the agent invents a balance when told the lookup failed. It cannot see that your refund API answers 200 when it fails. Check 4 is yours to run.

The short version: never let the sentence "done" be the evidence that it is done. Make the claim parseable, compare it with a record the model did not write, and prove the comparison can fail by pointing it at an agent built to lie before you point it at your own.