Automating contract review is five separable jobs (triage, clause extraction, playbook checking, redline generation, obligation extraction) and each has a different accuracy ceiling and a different failure cost. On ContractEval, a public benchmark running 19 models over 4,128 labelled clause-risk items from the CUAD corpus, the best proprietary model reached 0.644 F1 and the best open-weight model 0.540, against the 85 to 95 percent that product pages quote. Write the playbook first (standard position, ranked fallbacks, walk-away, trigger conditions) because an unwritten playbook produces an unevaluable system. Stage three must emit a citation to the exact source span rather than a summary: a public tally of court filings containing AI-fabricated citations passed 1,500 by mid-2026, with six figures of US sanctions in the first quarter alone. Auto-redline only where a deviation maps to an explicitly ranked fallback with a clean single-span match, and route cross-referenced defined terms, governing-law interactions, novel indemnity carve-outs and regulated data flows to counsel. Build the eval set from your own last 150 to 300 executed agreements per contract family and instrument recall on missed risk per family, not aggregate accuracy, because 92 percent across 10,000 agreements is 800 misses. Structured extraction runs at least as well self-hosted (0.842 micro F1 against 0.820 for the strongest frontier model, at roughly a tenth of the cost per document), which removes the last accuracy argument for shipping the corpus to a third party.
Every guide on how to automate contract review with AI quotes the same accuracy band: 85 to 95 percent. The closest published third-party measurement says otherwise. ContractEval, a public benchmark that ran 19 models over 4,128 labelled clause-risk items drawn from the CUAD corpus across 41 legal risk categories, put the best proprietary model at 0.644 F1 and the best open-weight model at 0.540.
Both can be honest at once. Vendor figures are usually measured on one curated contract family with a playbook tuned for it, and your paper is not that family. Teams lose money here not by buying a bad tool but by automating a stage that was never safe to automate, then finding the misses eighteen months later at renewal.
Adoption is no longer the constraint: a 2026 general counsel survey put generative AI use inside corporate legal departments at 87 percent, against 44 percent the year before. Evaluation is. What follows is the pipeline, what breaks at each stage, and where the model stops.
"Automate contract review" is five separable jobs, not one
Conflating them is the most common reason pilots stall. Each has a different accuracy ceiling and a different failure cost, so a pilot scored as one number averages a 0.84 task with a 0.64 task and produces a figure nobody can act on.
Write the playbook before you touch a model
An unwritten playbook produces an unevaluable system. If the position on limitation of liability lives in three lawyers' heads and differs in each, there is no gold label, nothing to score the model against, and the pilot ends in an argument rather than a number.
A machine-checkable entry has four parts: the standard position as exact language rather than a description, ranked fallbacks with substitute text for each rung drafted from language you have signed, the walk-away condition, and the trigger conditions that say which rung a counterparty clause activates.
Worked, for limitation of liability: standard is a mutual cap at 12 months of fees, carve-outs for confidentiality breach, IP infringement and gross negligence; fallback one 18 months; fallback two 24 months with a data-breach super-cap; walk-away any uncapped liability or any cap below fees paid. Triggers: a cap as a fixed sum rather than a multiple, a one-way cap, a carve-out list that omits confidentiality.
Do the same for indemnity, data processing, termination for convenience and auto-renewal. Auto-renewal triggers are the easiest and most valuable: a notice window under 60 days, a renewal term longer than the initial term, or an uncapped price escalator.
Then steal the framing from ContractNLI, which models review as document-level natural language inference: for each hypothesis, return entailed, contradicted or not mentioned, plus the evidence span. Not mentioned is a distinct answer from compliant. Systems that collapse the two pass silently on missing clauses, which is how a contract with no confidentiality survival period gets marked clean.
The five-stage pipeline: ingest, segment, check, redline, register
Every stage has one guardrail. Drop it and the next stage inherits the damage silently.
Stage 1: ingest and parse
The system reads in a fixed order: the executed instrument, then amendments in date order, then schedules and exhibits, then anything incorporated by reference. An amendment read before the base agreement produces clause-typing errors that propagate downstream. Four things break here: scanned counterparts where only the signature pages are images and nothing warns you, exhibit tables that flatten into token soup and take the payment terms with them, exhibits that are spreadsheets rather than prose, and terms incorporated by a URL that has since changed. Guardrail: page-level parse confidence with a hard fail, because a silent partial parse looks exactly like a clean contract with fewer risks. Our comparison of document parsing engines for RAG covers the current options. When documents disagree, encode the precedence rule rather than letting the model infer it: the amendment beats the base agreement, the later amendment beats the earlier, an executed counterpart beats a draft. A conflict between the body and a schedule is a finding, not something the model should quietly resolve.
Stage 2: clause segmentation and typing
Segment on document structure, not token windows. The numbered clause hierarchy is right far more often than a semantic splitter. Two failure modes need explicit handling: clauses that span a page break, and definitions. Definitions are not operative clauses, which is why generic pipelines drop them, and they change every operative clause that uses them. A "Confidential Information" definition excluding anything received from a third party guts a confidentiality clause that reads perfectly on its own. Rule: every defined term inside a flagged clause travels with it into stage three.
Stage 3: playbook check with span-level citation
Retrieve candidate spans, classify each against the playbook hypothesis, and emit entailed, contradicted or not mentioned along with the exact character offsets in the source file. The citation is a span, not a summary, and the reason is public: a tally of court filings containing AI-fabricated citations passed 1,500 by mid-2026, growing by five to six a day, with six figures in US sanctions in the first quarter alone. Every one started as an output nobody checked against a source. Retrieval narrows this without closing it: a preregistered academic evaluation of the dominant legal research platforms still found fabricated or misgrounded propositions in 17 to 33 percent of queries. Guardrail: when flagged text and retrieved span disagree, the item goes to a human with both shown.
Stage 4: fallback selection and tracked changes
Pick the rung, draft from precedent, and write tracked changes into the original file, each carrying a note naming the playbook position or the prior executed agreement behind it. Guardrail: never compose novel language for a clause type with no playbook position. Return it flagged as unreviewed. A model that quietly approximates a position you never took is worse than no output, because it looks like work product.
Stage 5: obligation and date register
After signature, extract to a fixed schema: obligated party, obligation text with its source span, trigger event, notice window in days, renewal and expiry dates, price escalators, reporting duties and cadence. Key it to the matter, not the file. This stage pays for the other four. Long-running contracting research puts average value erosion, the gap between what an agreement was supposed to deliver and what it did, near 9 percent of contract value, with the worst performers above 20 percent. Most of it leaks through obligations nobody tracked and renewals nobody diarised.
What published benchmarks actually say about accuracy
The failure mode is silence, not error. On ContractEval, one open-weight configuration answered "no related clause" on more than 30 percent of items that had one. A false negative here does not look like a mistake. It looks like a clean contract. Nor does scale fix it: the 8B model in that family scored 0.540, beating both the 4B at 0.411 and the 14B at 0.473.
Detection without justification is not review. A public discrepancy benchmark planted 10 categories of defect into more than 7,500 contracts and found that leading models miss the subtle ones and often cannot say why a defect is one legally. A flag your reviewer cannot verify in ten seconds costs more than it saves.
Do the arithmetic before you sign anything. Ninety-two percent across ten thousand agreements is eight hundred misses. If forty are auto-renewal dates and six are uncapped indemnities, the tool cost more than it saved. Instrument recall on missed risk, per clause family, never aggregate accuracy.
One number from that NDA head-to-head deserves rescuing. The figure worth keeping is not the 94: it is the human spread, 67 percent at the bottom and 94 at the top on the same five documents. Variance is what automation actually solves. The floor rises; the ceiling was already fine.
| Task | Best third-party result | Typical product-page claim | Unattended? |
|---|---|---|---|
| Clause-level risk identification (CUAD, 41 categories) | 0.644 F1 proprietary, 0.540 open weight | 85 to 95% accuracy | No |
| Structured field extraction (26 fields, SEC contracts) | 0.842 micro F1, self-hosted domain model | 94%+ extraction accuracy | Yes, above a confidence floor |
| Planted-defect detection (CLAUSE, 7,500+ contracts) | Misses subtle defects, cannot state the legal reason | "Catches what humans miss" | No |
| NDA issue spotting (vendor head-to-head, 5 NDAs) | 94% automated vs 85% human, spread 67 to 94 | 85% faster review | First pass only |
| Retrieval-grounded legal answers | 17 to 33% of queries fabricated or misgrounded | "RAG removes hallucination" | No |
Where the model must stop: the escalation matrix
Route by rule, not by a confidence score the model produces about itself: calibration on legal text is poor enough that a threshold gives the feeling of control without the fact of it. The point of the other four stages is that the reading gets done in minutes, so the judgment time is protected.
The audit record has to survive being read eight months later by someone who was not there. A defensible one carries model and prompt version, playbook version, the retrieved span with character offsets, the classification, the proposed change, the reviewer, the decision and the timestamp, written immutably. If you cannot reconstruct why clause 11.3 was accepted, you have a log, not a record. Our guide to human-in-the-loop approval covers the gate mechanics.
| Situation | Route | Why |
|---|---|---|
| Matches the standard position, single clean span | Auto-accept, logged | Nothing to decide |
| Deviation maps to a ranked fallback, single span, no cross-reference | Auto-redline, reviewer confirms | Position decided in advance |
| Playbook requires a clause and it is absent | Auto-insert from precedent, reviewer confirms | Absence is a finding, not compliance |
| Clause modified by a defined term or cross-reference elsewhere | Counsel | Span checking cannot see the interaction |
| Governing law or jurisdiction interacting with another flag | Counsel | Enforceability is jurisdiction-dependent |
| Novel indemnity carve-out or liability construction | Counsel | No prior position to reason from |
| Regulated data flows: personal data, health data, transfers | Counsel | Exposure sits outside the paper |
| Playbook has no written position on the clause type | Counsel, flagged unreviewed | Approximating a position you never took is the worst output |
| Retrieved span and flagged text disagree | Counsel, both shown | Highest-yield error signal |
| Incorporated online terms changed since last execution | Counsel | The instrument changed without anyone signing |
This is not workflow automation, and treating it as such breaks the build
Workflow automation and RPA assume two things: the input arrives in a fixed shape, and the next step is known before the current one runs. Contract review has neither property, and that determines what you should buy.
The input is a pile of mixed-format documents: an executed PDF, a scanned counterpart, a Word counter-draft with someone else's tracked changes in it, side letters, schedules that are spreadsheets, terms incorporated by a link. Nothing normalises that into fields.
The next step depends on what the last step found. A change-of-control trigger sends the system to the counterparty's group structure and the assignment clause. A data processing addendum sends it to the transfer mechanism. You cannot draw that as a flowchart in advance, because the branch set is a function of the document.
And the output is a work product a senior signs: a redline, a ranked issues memo, a fallback ladder, and a note on what is being conceded and why. Not a status field. So the unit of evaluation is the work product, not the step. That is the shape our contract associate agent is built for: it pulls the prior agreements with that counterparty, reads the instrument with its schedules and incorporated terms, checks each position against the playbook and against what your counsel accepted before, then produces all four artefacts together. For where this work sits in an operating model, see our AI for business pillar.
A clause library priced per seat is a different product from a system that carries the file across steps and defends its own output. Evaluate them differently.
Build your eval set from contracts you have already closed
Vendor accuracy numbers never transfer to your paper. The only number that means anything is measured on your contracts, against your playbook, by your reviewers, and the material is already in your archive. The difference between the counterparty's draft and the executed version is a labelled negotiation outcome. Mine that.
Our guide to building evaluation datasets for business AI covers the labelling workflow.
Why this runs inside your network, and how to roll it out
Third-party paper usually arrives under confidentiality terms restricting disclosure to processors and requiring consent before a new subprocessor touches it. Sending the counterparty's draft to a hosted model can breach the very agreement being reviewed. Add privilege on anything touching a dispute, and data residency for EU counterparties, and the deployment question answers itself.
The accuracy argument for shipping the corpus out has closed too. A May 2026 study compared a self-hosted, legal-domain small model against five frontier models on 26 structured fields from held-out SEC contracts. The self-hosted model came out ahead on aggregate F1 (0.842 micro against 0.820 for the strongest frontier model) at roughly $0.018 per document batched against $0.149 to $0.456, on two NVIDIA H200 GPUs. A 78 to 97 percent cost reduction with no accuracy penalty, consistent with the pattern in why specialized 7B models outperform frontier models in production.
Deployment shape: a mirrored clause library and precedent bank inside your network, private inference, retrieval restricted to sources you already license, no open-web calls. We built this for a law firm where privilege made the boundary non-negotiable, written up as on-premise document AI at a law firm. The tradeoffs are in our cloud versus on-premise comparison, and the law-firm view in how law firms are actually using legal AI.
Instrument four metrics from day one: first-pass turnaround, deviation catch rate against the frozen holdout, escalation rate, and the share of redlines accepted without edit. The fourth tells you whether the fallback ladder is drafted from real precedent or from plausible-sounding language.
If escalation rate climbs rather than falls after six weeks, the problem is not the model. The playbook has no position on the clauses your counterparties actually push on. Write those down and re-run the holdout. That is the whole loop.
A rollout that survives quarter two
Sequence by contract family, not by department. Each family gets its own playbook, eval set and go-live decision.
| Order | Family | Runs unattended first | Stop condition |
|---|---|---|---|
| 1 | Inbound NDAs | Triage, typing, playbook check, redline to ranked fallbacks | Missed-risk recall below 95% on the holdout |
| 2 | Vendor paper (standard forms) | Triage, typing, playbook check; redline reviewed | Any missed walk-away clause, ever |
| 3 | Customer paper on your form | Deviation detection against your template | Redline acceptance below 60% |
| 4 | Bespoke negotiated agreements | Obligation register only, post-signature | Escalation above 50%: playbook not ready |
FAQ
Quick answers to the questions this post tends to raise.
