Why Your AI Agent Forgets the Rule You Gave It (and How to Test Context)

You opened the conversation with one rule. Always attach the ticket id. Never quote a delivery date without checking the depot. Do not promise a refund without an order number. The agent obeyed it immediately, obeyed it at turn ten, obeyed it at turn thirty — and somewhere around turn fifty it stopped, without a word. No error, no warning, no line in the log saying the rule had aged out. The agent just carried on sounding exactly as confident as before, and the only person who noticed was the customer who got the wrong answer.

A context window is a capacity, not a memory

Model cards advertise context in tokens, and it is easy to read that number as a promise: two hundred thousand tokens, therefore it remembers everything. But capacity is not retention. What actually matters in production is whether a specific instruction, given once at the start, is still shaping behaviour after the thread has filled with noise — and that is an empirical property of your prompt, your conversation shape and your model, not a spec-sheet figure. A large window makes forgetting less likely. It does not make it observable.

This is why context failure is so expensive. Truthfulness failures are loud once you look; a fabricated citation is wrong on its face. A context failure produces output that is well-formed, fluent, and wrong only relative to a rule that scrolled off the top of the conversation. Nothing in the response advertises that it has drifted.

Three failures that look identical from outside

In the Clinic's battery, D5 Context Window Management is three separate tests, because an agent can pass one and fail the next — and the fixes are different:

• —long-thread-recall — a fact planted early is needed late. Needle-in-thread recall probed at turns 20, 40 and 60. Failure mode: the agent invents a plausible substitute rather than admitting the fact is gone.
• —instruction-retention — a rule given at turn 1 is checked at turn 50. Failure mode: the rule is silently dropped while the tone and confidence stay identical.
• —context-compression — behaviour when the window actually fills. Failure mode: the agent truncates, then contradicts something it said earlier without flagging the contradiction.

Notice that all three fail quietly. That is the defining property of this dimension: there is no exception, no refusal, no partial-output marker. The agent that forgot your rule and the agent that never had it produce the same transcript.

A probe you can run in five minutes

The important design choice below is that the harness owns the answer key. The rule and the planted fact are written by the test, not by the agent, and the check is a mechanical string test rather than a judgement call. If you let the model tell you whether it remembered, you are asking the defendant to sign the verdict.

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-opus-4-8"

# The harness owns the truth. The agent never supplies its own answer key.
RULE = "End every single reply with the ticket id in square brackets: [TCK-4417]."
NEEDLE = "The customer's replacement unit ships to the Rotterdam depot, not Antwerp."
NEEDLE_Q = "Which depot does the replacement unit ship to?"
NEEDLE_ANSWER = "rotterdam"

FILLER = [
    "Can you summarize the last message?",
    "What timezone should I use for the callback?",
    "Draft a one-line status note for the customer.",
    "Is there anything else you need from me?",
    "Rephrase that more briefly.",
]


def build_thread(depth):
    """Rule at turn 1, needle at turn 3, then `depth` turns of filler."""
    msgs = [
        {"role": "user", "content": f"You are a support agent. Standing rule: {RULE}"},
        {"role": "assistant", "content": "Understood. [TCK-4417]"},
        {"role": "user", "content": f"Case note: {NEEDLE}"},
        {"role": "assistant", "content": "Noted. [TCK-4417]"},
    ]
    for i in range(depth):
        msgs.append({"role": "user", "content": FILLER[i % len(FILLER)]})
        msgs.append({"role": "assistant", "content": f"Acknowledged ({i}). [TCK-4417]"})
    msgs.append({"role": "user", "content": NEEDLE_Q})
    return msgs


def probe(depth):
    reply = client.messages.create(
        model=MODEL,
        max_tokens=256,
        messages=build_thread(depth),
    ).content[0].text

    return {
        "depth_turns": depth,
        # long-thread-recall: did the fact from turn 3 survive?
        "recall": NEEDLE_ANSWER in reply.lower(),
        # instruction-retention: is the turn-1 rule still being obeyed?
        "retention": reply.rstrip().endswith("[TCK-4417]"),
        "reply": reply.strip()[:120],
    }


for depth in (5, 20, 40, 60):
    r = probe(depth)
    # Report both axes separately. An agent can remember the fact and forget
    # the rule, or obey the rule and invent the fact — these are different bugs.
    print(
        f"turns={r['depth_turns']:>3}  "
        f"recall={'PASS' if r['recall'] else 'FAIL'}  "
        f"retention={'PASS' if r['retention'] else 'FAIL'}  "
        f"| {r['reply']}"
    )

Run it and you get a small table with two independent columns. Recall and retention come apart more often than people expect: an agent will happily remember the depot and forget the formatting rule, because the fact was interesting and the rule was structural. Grading them as one number hides exactly the distinction you need in order to fix anything.

What this probe cannot tell you — including when we got it wrong

A probe that returns four rows of PASS is not proof your agent is safe past sixty turns. It is proof that this rule, in this conversation shape, survived. Change the filler from short acknowledgements to long tool dumps and the same agent can fail at half the depth. Depth in turns is a proxy; tokens are the real axis, and a probe that fixes one while varying the other is measuring a slice.

We are stating that plainly because we have been caught by the neighbouring mistake in our own grader, on this exact dimension. In one production scan, our D5 instruction-retention test came back with no finding at all — the judge returned nothing — and instead of recording that as missing evidence, the harness wrote a hard zero and stamped it "evidence: sufficient." The report then named D5 the agent's top risk. The dimension with the least evidence became the headline, and the number was our invention, not a measurement.

A score with no finding behind it is not a low score. It is not a score at all — and publishing it as one is the exact failure the tool exists to catch.

The bug is open on our own board and the fix shape is not complicated: an empty or fallback judge response has to be treated as absent evidence, excluded from the dimension average, and if a dimension ends up with no evidenced tests it must be marked NOT TESTED rather than scored. We mention it here because the honest version of "how to test context" includes the part where the tester fails, and because a reader deciding whether to trust a grade deserves to know which failure modes the grader itself has.

What to actually do about it

Context retention is one of the few reliability problems with genuinely boring engineering answers, most of which do not involve a bigger model:

• —Re-inject the rules. Do not rely on turn 1 surviving; restate hard constraints on a cadence, or pin them into a system prompt that is rebuilt every request.
• —Checkpoint the facts. Extract the small set of decisions and constraints into a structured scratchpad the agent reads, instead of hoping they stay in the transcript.
• —Guard the window. Decide explicitly what gets dropped when the thread fills, rather than letting silent truncation choose for you.
• —Test at the depth you actually run at. If your real conversations reach eighty turns, a twenty-turn eval tells you almost nothing about production.
• —Measure recall and retention separately, and record which one degraded — they have different fixes.

If you would rather see the whole picture than one dimension, the Clinic runs eighteen tests across six dimensions and returns an honest A–F — including marking any dimension the evidence could not support as NOT TESTED and excluding it from the grade, rather than filling the gap with a confident zero. Five scans a month are free and need no signup. Our guide on how to test an AI agent covers the wider framework; this dimension is only one axis of six, but it is the one that fails most quietly. The other five, and the order worth fixing them in, are laid out in why agents fail in production.