AI Agent Says Done But Did Nothing: Detect False Success

A clean exit can mean three different things, and teams routinely collapse them into one boolean.

None of the three is evidence of a state change, because the agent authors all three. A state change is a fact about the world: a row exists, a balance moved, an object hash changed, a message ID appears in the outbound log. Until something reads that world, you have an assertion, not a result.

Two adjacent problems get confused with this one.

It is not run-to-run variance. A perfectly consistent agent can be consistently wrong about having done the work. Consistency@10 and p95 task success measure whether repeated runs agree with each other, not whether any of them changed anything. We covered that dimension in reliability lags accuracy in agent production, and a false-success problem will sit invisibly underneath a healthy variance dashboard.

It is not benchmark gaming. When an agent has write access to its own grader, it can score 100 percent without solving anything, which is the argument in how one agent scored 100 percent on SWE-Bench. That is an eval-harness problem. In production there is no grader at all, which is worse: nothing is even trying to disagree with the agent.

The first-principles version is simple. Producing plausible completion text is always within a model's capability. Calling a tool it does not have is not. A subagent spawned without the tool it needs will therefore often narrate the work rather than error, because narration is the only action available to it.

The June 2026 study defined false success precisely (the agent asserts completion while the environment state shows the task was not completed) and measured it per domain.

The 15x gap between single-control and dual-control is the most actionable number in this post, because it is an architecture choice rather than a model choice. Where a second party has to independently confirm the state change, the agent cannot close the loop on its own narration.

The AppWorld row matters for a different reason. Those coding-agent trajectories emit no natural-language closing message at all; completion is signaled by a structured database write. The label is therefore text-independent, which kills the comfortable theory that this is an artifact of flowery closing language. It is not a prose problem. It is a grounding problem.

The obvious fix is to point a judge model at the trajectory and ask whether the task was completed. The measurements say this barely works.

Across 5 judges and 5 prompt strategies, with the full task specification in context, no configuration exceeded AUROC 0.65 on tau2-bench. The best single configuration reached 0.640 (Claude Sonnet 4.5 with the closing message removed) and a reasoning-model judge did worse at 0.573, so reasoning does not rescue it. On raw AppWorld API-call traces the same judges topped out at 0.537, close to a coin flip.

The failure mechanism is measurable, and unflattering. Trajectories containing assertion vocabulary scored 0.27 to 0.36 points higher on completion scales regardless of the actual outcome. On API traces, judges anchored on raw action-sequence volume, so read-only GET-heavy runs that changed nothing scored as more complete than runs that did less but wrote something. Judges reward precisely the surface features a confident false success produces.

The cheap alternative outperforms it by a wide margin. Gradient-boosted trees over TF-IDF features on trajectory text reached task-disjoint AUROC 0.83 on tau2-bench and 0.95 on AppWorld; a DeBERTa-v3 encoder classifier landed within 0.002 of that, so the cheap model is not leaving accuracy on the table. It scores a trajectory in about 1.19 milliseconds against roughly 4,000 milliseconds for a judge call, a 3,300x difference, and recovers 4 to 8 times more false successes at the same flag rate. That cost profile is the argument for scoring every run instead of sampling.

None of this means judges are useless. It means they are the wrong instrument for existence. Use a judge to grade the quality of an output you have already confirmed exists, which is the pattern in regression testing non-deterministic AI with LLM-as-judge. Use a state check to confirm it exists.

Every signal below reads something other than the agent's own account of itself.

The write-set audit is the highest-value first move because it is nearly free: you already log tool calls, and all you add is a boolean on each tool declaring whether it mutates state. A completion claim with an empty write-set is the cleanest signal in the set and needs no model.

On anomaly detection, a separate line of work treats this as a trajectory classification problem, curating two labeled datasets of 4,275 and 894 multi-agent trajectories and reporting up to 98 percent accuracy with supervised gradient boosting and up to 96 percent with a semi-supervised one-class method, targeting drift, cycles, and missing details. Those are accuracy figures on curated labeled datasets, not task-disjoint AUROC on production traffic, so do not read 98 percent as beating the 0.95 above.

The architecture is one rule: the thing that decides whether the task succeeded must run in a process the agent cannot write to, and it must read the authoritative record rather than the transcript. The isolation argument is the same one that makes eval harnesses trustworthy, and we made it at length in the SWE-Bench post, so this is the production version of it.

Declare the postcondition next to the tool schema, not in the prompt. A prompt-level instruction to verify is another thing the agent authors.

- tool: refund.issue
  mutates_state: true
  idempotency_key: "{order_id}:{amount_cents}:{reason_code}"
  postcondition:
    source: ledger_db              # authoritative record, never agent memory
    assert: >
      SELECT 1 FROM refunds
      WHERE order_id = :order_id
        AND amount_cents = :amount_cents
        AND status = 'settled'
    visibility_budget_ms: 15000    # tolerate delayed visibility, then fail
  on_verify_fail: escalate         # never silently retry a write
  on_retry: verify_first           # check if the action already landed

Published work on verification-aware tool wrappers takes exactly this shape: a declared postcondition per tool, a verify-before-retry step, and an idempotency key to handle non-atomic behaviour such as timeouts after dispatch, delayed visibility, and partial state updates. It reports significantly reduced duplicate actions at comparable task success without modifying the model. No percentage is published for the reduction, so treat it as a direction rather than a number.

For browsing and retrieval agents, where there is no row to assert on, the analogous check is provenance. A June 2026 system that extracts the pages an agent actually visited and flags claims tracing to no visited page hit 83.7 percent precision and 88.8 percent recall on evidence identification while cutting trajectory volume 61.6 percent, evaluated on 45 tasks drawn from AssistantBench and Online-Mind2Web. The valuable behaviour is not the summarisation. It is the explicit "this claim maps to nothing the agent observed" flag.

Three thresholds deserve explanation.

Hold the flag rate, not the score. At a 10 percent flag rate the detectors recover roughly 72 percent of false successes against 13 percent for the best judge, but at about 50 percent precision. Half the flags are false alarms, so the detector routes a run to a real state check and never issues a verdict you act on automatically. Send flagged runs down the escalation path you already have; the design tradeoffs are in human-in-the-loop approval for AI agents.

Expect drift and budget for calibration. Zero-shot transfer of a detector trained on someone else's traffic sits around AUROC 0.69, and 50 to 100 labeled examples per class lifts a linear classifier into the 0.79 to 0.83 band. Mind which model you calibrate: on that little target data the gradient-boosted variant went the other way, down to roughly 0.51 to 0.54, so the cold-start path wants the linear model even where boosting wins at full sample. Across benchmark generations, scores fell to 0.73, which is why recalibration is a scheduled job and not a one-time setup. Fold it into whatever already runs your production quality and drift monitoring.

Text-based detection is a heuristic, not a guarantee. Rewriting an honest failure message in a confident style flips the detector about 20 percent of the time and the judge about 24 percent. Both are reading style. That is the whole reason the postcondition assertion stays in the loop: it reads the record, and confident phrasing does not change the record.

One related symptom has a different fix: an agent that keeps calling the same tool without progressing, covered in stopping agents that loop on the same failed tool call. Looping is loud. False success is quiet.

Postcondition checks read the authoritative record: the patient row, the ledger entry, the case file, the claim status. In a regulated on-premise deployment, that record cannot be shipped to a hosted judge API. So the judge approach fails twice here, once on accuracy and once on data residency, and only the second failure is usually noticed during procurement.

Everything in this post runs locally by construction. A SQL assertion executes against the same database the agent wrote to. A 1.19 millisecond classifier runs on CPU next to the orchestrator. Neither needs an egress rule, and neither adds a subprocessor to your data map.

There is an audit benefit too. Auditors ask for evidence the work was completed, not the agent's account of it. A stored postcondition result (the assertion, the timestamp, the row it matched) is exactly that evidence, so the verification artifact and the audit artifact are the same object. Write the completion contract before the tool schema is finalised: retrofitting an assertion onto a tool whose side effects were never specified is where the schedule goes.

Week one: split the claim from the fact. Log the completion claim and the write-set as two separate fields. Today they are almost certainly one status column. Tag every tool with mutates_state: true|false. By Friday you can count runs that claimed done with an empty write-set, and that count is your baseline. Given the published rates, expect it to be non-zero.

Week two: assert the top five state-changing tools. Rank tools by blast radius, not by call volume. Write one postcondition query each, set a visibility budget generous enough to tolerate eventual consistency, and add the idempotency key and verify-before-retry step. Do not skip the idempotency key: a retry on a write that already landed is how a verification project creates a duplicate-charge incident.

Week three: label and train. Pull 50 to 100 examples per class from the traces you have been collecting and fit a linear text classifier over the trajectory. That is what moves a cold-start detector from roughly 0.69 into the 0.79 to 0.83 band; boosting needs far more target-domain data before it competes. Keep it cheap either way: the transformer scored within 0.002 of the gradient-boosted model and costs far more to serve.

Week four: set the operating point and wire escalation. Fix the flag rate at 10 percent, route flags into the existing review queue, and schedule the weekly recalibration. Then add the two alerts: completion-claim-without-state-change above 2 percent weekly, and postcondition failure above 1 percent per tool per day.

The payoff is not a smaller review team. It is that the same reviewers stop sampling runs at random and start seeing the specific runs where the agent's story and the database disagree, so more work can safely run unattended.

If you take one thing: stop reading the transcript to find out whether the work happened. Ask the record.