Industry benchmarks put rule-based transaction monitoring false positives at roughly 90 to 95 percent, with published estimates putting a routine alert at 20 to 30 minutes of analyst time and a complex one at 60 to 90 minutes. You cannot fix that by raising thresholds, because US banking agencies confirmed in 2021 that model risk management guidance applies to BSA/AML systems, which makes any change to what alerts, scores or closes a validated model change subject to above-the-line and below-the-line testing. The lever that actually moves is the reading: an analyst spends most of that half hour assembling a case file across core banking, KYC, screening, prior alerts and scanned documents, and only a few minutes deciding. Build the pipeline in five separately validated stages (evidence bundling, baseline comparison, bounded network expansion, hypothesis testing, narrative draft), enforce an output contract where every figure carries a record identifier and supporting, contradicting and missing evidence sit in separate fields, and compute every total in code rather than in the model. A 2026 paper on explainable AML triage reports citation validity of 0.98 against an Escalate F1 of only 0.62, which is the whole argument for keeping disposition with a person: grounding fixes auditability, not judgment. Run shadow mode first, allow auto-close in exactly one narrow alert category, and sample every auto-closed alert below the line at a stated rate that never reaches zero.
The largest Bank Secrecy Act penalty on record, roughly 3.1 billion dollars in 2024, did not turn on analysts making bad calls. It turned on coverage. Roughly 92 percent of transaction volume, around 18.3 trillion dollars, never reached the monitoring system at all between January 2018 and April 2024. Nobody triaged those transactions badly. They were never alerted on.
Hold that frame when you evaluate AI for AML alert triage, because the usual pitch points the other way. The pitch is that AI cuts alert volume. The failure mode regulators punish is that your alert volume was already too small in the wrong places. Meanwhile the alerts you do generate are overwhelmingly noise: industry benchmarks put rule-based transaction monitoring false positives at roughly 90 to 95 percent, with published estimates putting a routine alert at 20 to 30 minutes of analyst time and a complex one at 60 to 90 minutes. Regulator-published data puts US suspicious activity report volume near 4.7 million in fiscal 2024, close to 13,000 a day. Bank filings rose almost 8 percent year over year in 2025, past 2.19 million for depository institutions alone.
The queue grows faster than any team, and the mechanism that shrinks it is the same mechanism that creates coverage gaps. So the question is not how to generate fewer alerts. It is what an AI system may legitimately do to those 30 minutes, what it must never touch, and what evidence you retain so the arrangement survives an exam.
What a 95% false positive rate actually measures
Three numbers get quoted interchangeably and they are not the same measurement.
A 95 percent false positive rate is not proof the rules are broken. It is the arithmetic of high recall against a very low prior. If reportable behavior appears in one account per thousand, a detector tuned to catch most of it fires mostly on legitimate activity. That is the correct engineering outcome. The mistake is treating it as an efficiency metric with an obvious fix.
Why raising the threshold is a model risk event, not a tuning exercise
Standard tuning practice tests thresholds 10 percent above and below current settings. Above-the-line testing confirms the alerts you generate are accurate. Below-the-line testing exists for one reason: to prove nothing productive is slipping under the line. Every threshold change answers both. US banking agencies clarified in 2021, in consultation with FinCEN, that model risk management guidance applies to systems supporting BSA/AML compliance. Anything that scores, ranks, suppresses or closes an alert sits inside that perimeter and needs documented development, independent validation, outcomes analysis and explainable inputs and outputs. A prompt change is a model change. A retrieval index rebuild is a model change. So the lever is not fewer alerts. Do not tune the noise away. Tune the cost of clearing an alert and the quality of the disposition record instead, which is a different engineering problem with a far lighter regulatory footprint. On which layer should own which decision, our framework for choosing between AI, rules and humans covers the general version of the call.
| Metric | Definition | Typical value | What it tells you |
|---|---|---|---|
| False positive rate | Alerts closed without escalation, over alerts generated | 90 to 95% | Rules, thresholds and base rate together |
| Alert-to-SAR conversion | Filings over alerts generated | 1 to 5%, well-tuned programs 8 to 15% | Whether escalation is finding anything filable |
| Productive alert rate | Alerts producing any investigative value: risk-rating change, relationship review, enhanced due diligence trigger | Rarely measured | Whether monitoring is working at all |
Where the analyst hour goes: assembling the case, not making the call
Watch what consumes the 30 minutes. An analyst opens, in rough order: the alert payload and rule parameters; 12 to 13 months of core banking history across every account the customer holds; the KYC and CDD file (expected activity, source of funds and wealth, industry code, beneficial ownership, risk rating and its last review date); sanctions, PEP and adverse media hits with their prior dispositions; wire messages and ACH addenda; prior alerts and SARs on the customer and related parties; scanned statements and correspondence in a separate imaging system; and free-text relationship notes, frequently the only place the real explanation lives.
The judgment step at the end is maybe five minutes. Everything before it is reading and transcription. AWS has published a reference architecture for AML alert triage reporting investigation time dropping from 30 to 90 minutes down to under five. That is a vendor figure with its own caveat that results vary, so hold it loosely, but the mechanism it describes is exactly this: the assembly collapses, the decision does not.
Note what is not on that list. Nothing requires the open web. Every source is a system the institution already runs or already licenses, and all of it is nonpublic customer information, so the cost and security tradeoffs of on-premise versus cloud AI resolve unusually cleanly here: no cloud convenience argument survives the vendor risk review.
This is not a workflow tool and it is not an RPA job
The distinction determines what you buy and what you build. Three properties put alert triage beyond workflow automation. The input is a pile of mixed-format documents. Not an API response with a schema. A tax return, a scanned statement from another institution, a relationship note, a wire with a truncated beneficiary field. RPA assumes stable interfaces. Here the interface is a PDF someone faxed. The control flow depends on what the last step found. A wire into a documented trade corridor changes which KYC field becomes decisive. A structuring pattern at a cash-intensive business sends you to the merchant category and the deposit slips. A screening hit on a beneficial owner sends you to the ownership chart. You cannot draw the flowchart in advance, because the flowchart is the investigation. The output is a work product a senior person signs. Not a state transition in a case management system. That combination separates junior-professional work from clerical work, and it is why extraction accuracy is the floor for everything above it. Our on-premise document AI build for a US commercial lender auto-posts 73 percent of extracted fields above a 98 percent precision floor with a per-field audit trail, which is the prerequisite for any triage layer sitting on scanned inputs.
The five stages of an AI-assisted triage pipeline
Build it as five stages with hard boundaries, each separately validated, so that when a validator asks what the system did and why, the answer is a stage record rather than one monolithic model call.
The ordering is not decorative. Bundle evidence first and score later, never the reverse, because a system that ranks before it retrieves will rank confidently on an incomplete file, and you cannot tell that apart from a genuine low-risk finding. This staged shape is what the financial crime analyst agent we build for institutions implements: it assembles the case across transaction, KYC and screening systems, tests the pattern against the customer's own baseline and their peer group, drafts the narrative with every fact sourced, and stops at a recommendation.
| Stage | Reads | Produces | Not permitted to | Validated by |
|---|---|---|---|---|
| 1. Evidence bundling | Alert payload, core banking, KYC and CDD, screening hits, prior alerts and SARs, imaged documents | Citation-indexed bundle: every record with an ID, every document with page and version | Summarize, score, rank or silently drop anything | Retrieval completeness against a per-alert-type checklist; a missing source is a named gap, not silence |
| 2. Baseline comparison | Bundle, onboarding activity profile, 13 months of self-history, peer segment statistics | Deviation statements with the compared figures and their sources | Invent a peer group or compare to a global threshold | Recomputation of every figure in code from retrieved rows |
| 3. Network expansion | Counterparties on the alerted transactions, account graph, prior related-party links | Bounded subgraph with a documented stop rule and per-edge evidence | Exceed the hop, node or degree budget, or traverse a hub node | Determinism test: same alert, same subgraph, plus a replayable log |
| 4. Hypothesis testing | Bundle, baseline, subgraph | Ranked hypotheses, each with supporting, contradicting and missing evidence in separate fields | Assert a conclusion whose refuting evidence was never retrieved | Counterfactual checks: perturb a fact, confirm the conclusion moves |
| 5. Narrative draft | Everything above | Structured draft: activity described, every figure cited to a record ID, gaps stated, recommendation | Generate a number it was not handed, or take a disposition action | Citation validation on every figure, then human review |
Test the hypothesis against the customer's own baseline
Global thresholds are what generate the 95 percent. A 9,500 dollar cash deposit means one thing for a salaried employee and another for a car wash whose onboarding file documents 40,000 dollars a week in cash receipts. Useful triage compares against three reference frames, in order:
Each hypothesis carries three separate fields, never one blended paragraph: evidence that supports it, evidence that contradicts it, and evidence that would settle it but is missing.
The third field is where most implementations fail. There is a difference between "no source of funds documentation exists on file" and "a source of funds document exists but OCR failed on it, or retrieval did not return it." A system that cannot distinguish those two states produces confident closures on incomplete files, which is precisely the below-the-line failure a validator hunts for. The rule we enforce: any hypothesis whose refuting evidence would come from a source the retrieval layer did not successfully read is ineligible for auto-close and routes to an analyst with the gap named.
Sources will also disagree. The KYC file says retail bakery, the transaction pattern says wholesale distribution. Do not let the model reconcile that. The disagreement is the finding. Put both records side by side with their identifiers and escalate.
How far to walk the counterparty network before you stop
One hop always. A second hop only under a stated trigger. Never unbounded. Unbounded traversal feels thorough and produces narratives no examiner can reconstruct, because almost every business connects to almost every other business within three hops through utilities, payroll processors and tax authorities. Set the budget before the first walk:
Hub nodes need handling, not just exclusion. That a customer pays a utility is not evidence. That eleven otherwise unrelated accounts share an obscure payment processor and a device fingerprint is. Keep hub edges in the bundle as context, mark them non-expandable, and never let a hub node form the connecting link in a stated hypothesis. On the retrieval side of the same problem, see our notes on GraphRAG implementation on an enterprise data platform.
What the model may decide, and what stays with the investigator
Write this table before you write any code. It is the document a validator will ask for, and the one that keeps scope from drifting.
Two rows are non-negotiable. The SAR decision never moves, in any category, at any accuracy level. And no alert with an incomplete bundle ever receives a published recommendation, because a recommendation on a partial file is worse than none: it anchors the reviewer. The general pattern, deciding which actions need an approval gate, is covered in our guide to human-in-the-loop approval for AI agents.
| Alert category | Evidence state | Permitted action | Human control | Retained evidence |
|---|---|---|---|---|
| Duplicate of a disposed alert: same rule, pattern, period | Complete, prior disposition retrieved | Auto-close, linked to the prior disposition | 100% review for one quarter, then 10% or higher | Prior alert ID, match rule, field-level diff |
| Rule hit explained by the documented profile, inside self-history bands | Complete | Recommend close, analyst confirms | Every recommendation reviewed | Bundle, deviation figures, analyst ID, timestamp |
| Screening hit with documented identity discrepancies (birth date, nationality, entity type) | Complete | Auto-close on the named discrepancy fields | Below-the-line sample at 5% minimum | Match score, discrepancy fields, list version |
| Deviation from the customer's own baseline, no documented explanation | Complete | Escalate with a drafted case file | Full second-line investigation | Bundle, hypotheses, counterfactual results |
| High-risk relationship: PEP, correspondent banking, MSB, trade finance | Any | Assemble and draft only, never close | Analyst decides every alert | Full bundle, no auto-action permitted |
| Retrieval incomplete, or sources contradict | Incomplete | Route to a human with the conflict named, no recommendation | Analyst resolves the conflict first | The conflicting records side by side |
| The SAR filing decision | Any | Never automated, in any category | Named investigator plus escalation chain | Filing record and full evidence trail |
The narrative output contract that survives an exam
The draft narrative is a structured object, not a blob of text. The contract has four clauses.
Every figure carries a record identifier. Transaction ID, document ID plus page, KYC field name plus version. A number without a pointer does not render.
No number is generated by the model. Totals, averages, counts, date ranges and percentages are computed in code from the retrieved rows and passed in as named facts, so the model composes prose around numbers it was handed. A mechanical validator then fails the draft if any numeric token does not match a passed-in fact. This one rule removes the whole class of hallucinated amounts, and it is the discipline described in how to fix broken RAG citations.
Supporting, contradicting and missing evidence stay in separate fields. Blending them is how a narrative reads as more conclusive than the file supports.
Counterfactual checks run before a human sees it. Perturb a retrieved fact, regenerate, confirm the conclusion changes. If the rationale survives removal of the fact it claims to rest on, the rationale is decorative and the draft fails.
There is published evidence for both the value and the limit of this. A 2026 paper on explainable AML triage with evidence retrieval and counterfactual checks reports citation validity of 0.98 and evidence support of 0.88 once grounding and counterfactual validation are added, against an Escalate F1 of 0.62 and PR-AUC of 0.75. Read those halves together: near-perfect citation validity beside a middling escalation F1 is this post's argument in one result. Evidence grounding fixes auditability, not judgment.
On the drafting architecture, a published agentic framework for AML compliance narratives decomposes the work into planning, crime-type detection, intelligence gathering and compliance validation agents, with dynamic memory, a privacy guard layer over sensitive data, and an agent-as-a-judge validator checking output in real time, while investigators stay in the loop to review and refine. The decomposition is worth copying. The paper publishes no benchmark numbers, so do not let anyone quote it as if it did.
One more result inverts the usual pitch. A 2026 production deployment paper on explainable money mule identification reports yield rising from 61 percent under rules to 89 percent, using a gradient-boosted model plus SHAP attributions turned into analyst-facing narratives, at a higher alert volume: 302 a month against 211. Volume went up. The win was alert quality, not suppression.
Validating a system that touches alert disposition
Federal agencies issued a joint statement in 2018 encouraging innovative BSA/AML approaches, and said explicitly that pilot programs which do not pan out will not automatically draw supervisory criticism. That is the cover for running triage in shadow mode before it touches anything. Use it.
Shadow mode first. Run the pipeline in parallel against live alerts without exposing output to analysts, then compare its recommendation against every actual disposition. Review each disagreement individually, in both directions. False closes matter far more than false escalations, and the disagreement log is the most valuable artifact you will produce.
Below-the-line sampling on auto-closed alerts, permanently. Start at 100 percent, fully reinvestigated by an analyst who does not see the model output. Step down on evidence, not on schedule, and never to zero. Any SAR-worthy finding in the sample suspends auto-close for that category until the cause is fixed and re-tested.
Champion versus challenger against the incumbent rules. The rules stay running. The comparison is not accuracy in the abstract, it is what each system does with the alerts the other one got wrong.
Drift monitoring on three surfaces. Input drift (new products, new corridors, a changed document format), output drift (a creeping auto-close rate is the first alarm to wire), and retrieval health (the share of bundles with a missing source degrades quietly and leads everything else).
Per-decision audit artifacts. Model version, prompt version, retrieval index version, the exact evidence bundle with record identifiers, the structured output, counterfactual results, the human action, and timestamps for each, retained on the same schedule as the underlying BSA records. If the artifact cannot be replayed to reproduce the recommendation, it is not an audit record. Our treatment of AI compliance in financial services covers the surrounding obligations, and the AI agents pillar collects the architecture patterns behind this pipeline.
Start narrow. Deploy evidence bundling alone, with no scoring and no disposition authority: it takes the largest bite out of the 30 minutes while sitting almost entirely outside the model risk perimeter. Add recommendations after 90 days of shadow comparison. Add auto-close for exactly one category, with full below-the-line review for a quarter. Done in that order, the same team clears a queue growing 8 percent a year, and the minutes they get back are the minutes they spend deciding.
FAQ
Quick answers to the questions this post tends to raise.
