HANDBOOK.md Benchmark: Why Agents Ignore Your Policy Doc

HANDBOOK.md is 65 agentic tasks across five domains: finance and accounting, HR, insurance, logistics, and medical billing. Not a subset, all five, and every one is a regulated vertical with an audit function.

The environment is a file workspace plus mock email, chat, calendar, issue-tracking, and commerce services exposed over the Model Context Protocol. Tasks draw on 10 fictional companies with 10 base handbooks, delivered as PDF, Word, and HTML. Grading runs on 824 deterministic criteria, checking both that required actions occurred and that prohibited actions did not.

The word doing the work is strict: a trial passes only if every criterion in its rubric is satisfied. Make the payment correctly, notify the right people, take one prohibited step, and the trial fails. A policy you satisfy 11 clauses out of 12 is a policy you violated.

Read the median column first. The typical handbook is 37 pages and 14.9K tokens. The 124-page figure that gets quoted is the single longest document, in logistics, and it is not the condition under which agents fail. They fail at 25 pages too. Finance and medical billing are where a stated control and an enforced one diverge most expensively: see AI compliance for financial services and HIPAA-compliant AI implementation.

Most agent benchmarks ask whether the agent finished. This one asks whether a binding document constrained it while it finished. A model can be excellent at the first and useless at the second.

The design decision that matters more: every task modifies one of the ten base handbooks, altering the specific rules and thresholds on which grading turns, so no two tasks share a policy. A model cannot pattern-match a memorized corporate norm, because the norm was mutated. That is why these scores sit so far below the ones quoted elsewhere, and we covered what happens without that discipline in why agent benchmark scores get hacked.

The harness is public under Apache-2.0. Price a re-test honestly: Docker, Python 3.13 or newer, uv, a container registry, API keys per model, and 65 tasks run four times each, roughly 260 trials per configuration.

The full published table, all 30 configurations, strict pass@1.

Three things a summary will not tell you.

This is not a smooth capability curve. The two Fable 5 rows sit 12.7 and 10.7 points clear of the next configuration, a gap wider than the whole band from third place down to tenth (23.5% to 13.5%). Any assumption that frontier models are roughly comparable on instruction following is unsupported here.

Strict grading is doing real work. The paper also reports a looser metric tolerating exactly one failed criterion, under which frontier scores roughly double. That clarifies rather than rescues: the authors' own writeup still puts the resulting top band below half, and a near-miss on a compliance rubric is a finding, not a rounding error. Same distinction we draw in measuring agent reliability in production.

Date-stamp everything. As of 2026-08-03 the live leaderboard carried 31 configurations, the paper's 30 plus Claude Opus 5, entering at 32.3% (adaptive/max) and 29.6% at default. The 36.2% top score still stood.

The paper's interpretation, close to its own wording: additional deliberation converts into rule compliance only when the underlying failure is a missed inference rather than a missed read. Policy failures are overwhelmingly missed reads. The rule was in context. It was not applied. No amount of thinking about the wrong thing fixes reading the wrong thing.

A cost note from the authors' writeup, approximate: GPT-5.5 spent roughly 13K tokens per trial at about a third the cost of Opus 4.8 at max effort (close to 60K), for a similar strict score.

The paper's failure analysis names four patterns, in its Section 6 wording:

The abstract words the third item differently, as agents losing rule details over long horizons, so treat those as two views of one list. The paper attaches no percentages to any mode: a pie chart of these four is invented.

Two cases from the authors' own writeup make the first two tangible. In an HR task, an email from a VP overrode a handbook requirement for written confirmation from one of two specifically named individuals: authority in the inbox beat authority in the document. In an expense task, the model confirmed the requester was a junior analyst signing off on his own entertainment bill, then approved it anyway.

Sit with the second one, because it defeats the intuition that better retrieval fixes this. The agent read the rule, ran the verification, got the disqualifying answer, and proceeded. It is adjacent to the pathology in agents that loop on the same tool call without progress: the tool layer behaves correctly and the decision layer does not consume it. The paper's own conclusion belongs in your design docs: the standing document does not function for current models as a persistent authority.

The obvious response is to bolt a classifier onto the output. The classifiers most teams reach for are the wrong tool, for a precise reason.

An ICML 2026 paper, Learning Efficient Guardrails for Compliance, introduces PolicyGuardBench: 59,997 policy-trajectory pairs drawn from a larger 314K-plus pool, of which 25,435 (42.4%) are violations, split 49,997 train and 12,000 test across five domains, covering full-trajectory and prefix-based detection.

Get the mechanism right, because it is easy to invert. The paper does not report that safety classifiers wave violations through. It reports that their outputs are highly skewed, often labeling nearly all inputs as either safe or unsafe, which the authors attribute to coarse-grained training objectives. That is a collapse toward one label in either direction, near chance both ways. Safety classification asks whether text is harmful; compliance classification asks whether a trajectory broke a specific clause.

We compared Llama Guard as a safety layer in AI guardrails compared; this is the compliance counterpoint. The purpose-trained result is not a green light either: reaching 0.9014 means training on your policy semantics, on your data, with a ceiling of roughly one miss in ten. In a payment or claims path, that is a triage filter, not a control.

Long handbooks and long horizons are clearly part of this. The authors' writeup reports a mean task length of 17 steps and roughly 30 tool calls. Our read of the context-rot literature in Chroma's long-context degradation analysis found accuracy dropping more than 30 points at positions 5 through 15 of 20 documents, 7.9% attributable to length alone across 18 models, and effective context at 50% to 65% of advertised under RULER. That is why we cap production context usage at 25% to 30% of the window, as set out in long-context LLM performance issues.

Now the honest part. HANDBOOK.md runs no controlled handbook-length ablation: it never holds the task fixed and varies document length, so nothing here establishes a causal length effect. Logistics carries the longest handbooks (median 72 pages, up to 124 pages and 79.4K tokens), the natural place to look, but the published results give no per-domain pass rate to check against.

Position decay is a contributing mechanism, not the whole story. Modes one and two are precedence and follow-through failures: the rule was present, recent, and read. Shortening the document makes mode three less likely and leaves the rest where they were.

The architecture follows from the modes: stop asking one model to remember, decide, and attest.

Buildtime: clauses become preconditions. Each enforceable clause compiles to a deterministic check bound to a tool and evaluated before the call executes. The model can request the action. It cannot perform it.

# Clause 4.2.1: expense reimbursements above $5,000 require written
# CFO or Controller approval; no self-approval.

@precondition(tool="expense.approve")
def clause_4_2_1(call, ctx):
    if call.amount_usd <= 5000:
        return Allow()
    if ctx.actor.id == call.requester_id:
        return Deny("4.2.1: self-approval blocked")
    if ctx.actor.role not in {"cfo", "controller"}:
        return Deny("4.2.1: approver role not on the named list")
    if not ctx.written_approval_on_file(call.request_id, ctx.actor.id):
        return Deny("4.2.1: no written approval artifact")
    return Allow()

That closes modes one, two, and three for that clause, not by persuasion but by deleting the code path: a VP email never reaches the enforcement point. This is the layer we evaluate in the MCP gateway comparison.

The prior art is citable. An EMNLP 2025 industry-track paper from IBM Research, Towards Enforcing Company Policy Adherence in Agentic Workflows, describes this exact split: a buildtime stage compiling policy documents into verifiable guard code associated with tool use, and a runtime stage running those guards before each agent action, evaluated on the tau-bench Airline domain. Its abstract reports encouraging preliminary results with no published adherence rate, so take the architecture, not a number. A 2026 ICML workshop paper, Autoformalization of Agent Instructions into Policy-as-Code, translates prompts, MCP tool descriptions, and policy documents into Cedar policy through a generator-critic loop, reporting on MedAgentBench that autoformalized policies cover substantially more of the source specification than hand-coded symbolic enforcement.

The strongest numbers come from a 2026 Carnegie Mellon study, Don't Make Models Guess Security and Safety: across 80 agent security and safety benchmarks, 74% of stated requirements are symbolically enforceable, and 85% of those benchmarks state no verifiable requirement at all. Two readings: most of your handbook is compilable, and nobody compiled it because it was never written in a form anything could check.

Runtime, out of band: a verifier that reads traces, not narration. The only control that touches mode four, because it does not read anything the agent wrote.

verifier: expense_approval_v3
input: execution_trace        # ordered tool calls, args, results
assertions:
  - id: "4.2.1-a"
    when: "any(call.tool == 'expense.approve' and call.args.amount_usd > 5000)"
    require: "exists(call.tool == 'identity.lookup'
                     and call.result.role in ['cfo','controller']
                     and call.result.id != approve.args.requester_id)"
  - id: "4.2.1-b"
    when: "any(call.tool == 'expense.approve')"
    require: "no_prior(call.tool == 'policy.check' and call.result.pass == false)"
on_fail: quarantine_and_page   # never trust the agent's own summary

Assertion 4.2.1-b answers the junior-analyst case directly: the check ran, it failed, the action happened anyway. The verifier catches that in the log even when the final report says otherwise.

This is becoming a procurement expectation, not just a research position. In February 2026 one large IT infrastructure services provider announced a policy-as-code governance layer translating organizational rules and regulatory requirements into machine-readable policies that govern how agentic workflows execute, citing research putting compliance concerns as a barrier to scaling technology investment for 31% of its customers.

One row per mitigation, one column per failure mode. Mode 1 is request overrides rule, 2 is check ignored, 3 is verification skipped, 4 is false compliance report.

Two cells deserve emphasis. The safety-classifier row is empty because near-chance accuracy on policy-trajectory detection is not partial coverage, it is noise. And the trace verifier is alone in column four, because it is the only control here that does not read what the agent wrote. Preconditions prevent, verifiers detect, and only the pair gives you an audit artifact that was observed rather than asserted.

The deployment procedure, in order:

This benchmark measured the deployment pattern almost everyone shipped in the last eighteen months, and it scored 36.2% at its best. That is not a model problem to wait out. It is a design that puts authority and enforcement inside the same generative process.

Particula Tech builds the other design for regulated on-premise deployments: clause inventory, compiled tool preconditions, trace-based verification, and a human gate sized to whatever is left. If your agent's compliance evidence is a paragraph the agent wrote about itself, that is the first thing to replace.