Text-to-SQL Accuracy: Why Benchmarks Lie in Production

The demo-to-production gap in text-to-SQL is roughly the distance between 86.6% and 10.1%, which is GPT-4o's execution accuracy on the academic Spider 1.0 benchmark versus its score on Spider 2.0's enterprise warehouses. The model did not get worse. The problem got real.

According to the Spider 2.0 paper (2024), the best code agent paired with o1-preview solved just 21.3% of Spider 2.0, versus 91.2% on the original Spider and 73.0% on BIRD. The GPT-4o baseline is the cleanest single-model illustration of the same effect: 10.1% on Spider 2.0 against 86.6% on Spider 1.0. What changed is not the language model but the databases. Spider 2.0's real applications often contain over 1,000 columns, and some BigQuery and Snowflake environments run past 3,000, whereas the academic schemas that produce the headline scores hold a few dozen columns at most.

This is not a niche academic observation. A widely cited real-database-gap analysis published on Beancount.io in June 2026 quantified the same cliff using BIRD and its enterprise successors, and the pattern held: the closer a benchmark gets to a production warehouse, the more the numbers fall. The 85%-ish figure in the vendor deck describes a world your warehouse does not live in. For teams building toward AI-driven data analysis that speeds up business insights, the leaderboard is a starting line, not a promise.

Spider, BIRD, and Spider 2.0 measure progressively harder versions of the same task, and the jump between them is exactly where benchmark accuracy stops predicting production accuracy. Reading a text-to-SQL claim without knowing which benchmark produced it is how teams end up promising the wrong number.

Spider 1.0 is the clean-schema baseline: cross-domain databases with tidy, well-named tables and unambiguous questions. It is largely solved. GPT-4o sits at 86.6% execution accuracy, and the historical ceiling is around 91.2%. If a vendor quotes "mid-80s to low-90s," this is almost certainly the benchmark behind it.

BIRD (NeurIPS 2023) raises the realism. It spans 12,751 question-and-SQL pairs over 95 real databases totaling 33.4 GB across 37 professional domains, and crucially it requires external knowledge to answer many questions. The original BIRD paper reported GPT-4 at 54.89% execution accuracy with curated external knowledge, falling to 34.88% without it, a roughly 20-point gap that is the whole story: the same model loses a fifth of its accuracy the instant you take away the business context. Humans scored 92.96%. BIRD is now largely tamed at the top of the leaderboard, where AT&T's AskData plus GPT-4o reached 81.95% on the test set in September 2025, so the real difficulty has moved to the enterprise benchmarks below. One more BIRD detail worth remembering when you build your own evals: about 49% of the financial-domain data points in BIRD contain some error, a reminder that dirty data is part of the task, not an exception to it.

Spider 2.0 (2024) is the enterprise-workflow benchmark, with the 1,000-plus-column schemas and multi-step warehouse tasks that produce the 21.3% best-agent and 10.1% GPT-4o figures.

The practical takeaway: match the benchmark's difficulty to your data before you trust its number, and if none of them look like your warehouse, the number to trust is your own. This is why building evaluation datasets that reflect your actual business questions matters more than any public leaderboard.

Real business questions are ambiguous, and ambiguity is where text-to-SQL falls apart: on BIRD-Interact, a frontier model answers well under 20% of enterprise questions in a single pass. The benchmarks above still hand the model a precise question. Your users will not.

BIRD-Interact, accepted at ICLR 2026 as an oral, was built to measure exactly this. It contains 600 tasks in its full set and 300 in its Lite set, all on PostgreSQL databases, and the questions are deliberately ambiguous business questions of the kind a real analyst gets in Slack. The results are sobering. On the full set, GPT-5 solved 8.67% of tasks in single-pass conversational mode and 17.00% when allowed to interact and ask for clarification. On the Lite set, the strongest reasoning models reach only around 24% in agentic mode and roughly 18% single-pass.

Two things fall out of those numbers. First, one-shot "chat with your data" is a trap: a model that guesses at what "active user" or "last quarter" means will confidently return the wrong table filtered the wrong way. Second, interaction roughly doubled GPT-5's score, from 8.67% to 17.00%, which tells you that letting the system ask a clarifying question is not a nice-to-have, it is a core accuracy mechanism. A large share of production text-to-SQL failures are not SQL-generation failures at all. They are the model resolving an ambiguous question the wrong way and never checking.

Text-to-SQL breaks at enterprise scale because schema linking, the step where the model maps words in the question to the right tables and columns, has to pick the correct handful out of thousands and gets it wrong. Schema linking is the quiet failure point that leaderboard schemas never stress.

Academic benchmarks use small schemas, on the order of 10 to 20 tables and a few dozen columns, where the entire structure fits comfortably in context and the right column is usually obvious from its name. Enterprise reality is different by an order of magnitude. Spider 2.0's databases often contain over 1,000 columns, and some BigQuery and Snowflake environments exceed 3,000. When the model has to choose revenue_net_usd_v3 over revenue_gross, rev_final, and forty other candidates, name-matching stops working and the query silently binds to the wrong column.

The newest evidence for this cliff is EntSQL. According to the EntSQL benchmark (arXiv, 2026), a benchmark for grounding text-to-SQL in long-context enterprise knowledge, the best evaluated system reached only 15.9% when long-form enterprise documents were supplied alongside the schema. EntSQL uses 1,066 aligned semantic examples across five business domains, and most of its questions require domain knowledge that simply is not present in the schema. That is the deeper problem: at enterprise scale, the definition of a metric lives in a data dictionary, a Confluence page, or an analyst's head, not in a column name. Handing a model 3,000 raw columns and hoping it infers your business is a losing bet, which is why teams increasingly reach for structured approaches like graph-based retrieval over an enterprise data platform to give the model navigable structure instead of a flat wall of columns.

A semantic layer fixes text-to-SQL by defining your metrics, dimensions, and join paths once, so the model generates against curated business logic instead of raw tables, and the accuracy jump is large. A semantic layer is a modeled abstraction that encodes what "revenue," "active user," and "enterprise segment" actually mean in your data, along with the correct joins to compute them.

The cleanest measurement comes from dbt Labs. According to dbt Labs (2026), on an unnormalized schema, text-to-SQL scored 64.5% while the semantic layer scored 72.7%, up from a 2023 baseline of 32.7% versus 60.5%. The more striking result is what minimal modeling does: after adding just three dbt models, text-to-SQL rose to the 84.1% to 90% range while the semantic-layer path reached 98.2% to 100% (Claude Sonnet 4.6 went from 90.0% to 98.2%, and GPT-5.3-codex went from 84.1% to 100%). A small amount of modeling, not a bigger model, produced near-perfect accuracy.

Snowflake's Cortex Analyst is the same idea in a product. According to Snowflake (2024), Cortex Analyst reaches roughly 90% or higher accuracy across customer evaluations and benchmark tests, and it is consistently close to 2x more accurate than single-shot SQL generation from state-of-the-art LLMs. Its documentation is explicit about why: semantic views supply descriptions, synonyms, join paths, and verified query examples, and generic AI solutions "often struggle" when given only a database schema. Snowflake also cites a Forrester finding that generic tools hit about 70% success at best on single-table queries and as low as 20% on multi-table queries, the raw-schema failure mode the semantic layer is designed to eliminate.

Across the enterprise data platforms we audit at Particula Tech, the single best predictor of production text-to-SQL accuracy is not which model you picked, it is whether a semantic layer exists at all. Teams that skip it and then blame the model are optimizing the wrong variable.

Without a semantic layer, promise 30 to 65% and expect as low as 10% on a warehouse the size of Spider 2.0's. With a modeled semantic layer plus verified examples, promise 90%-plus. Those two bands, not a single leaderboard figure, are what you should put in front of a stakeholder.

The anchors are all primary-sourced. The raw-schema band runs from dbt's 32.7% (2023) and 64.5% (2026) up through GPT-4o's 10.1% floor on Spider 2.0-scale warehouses, because a large, messy schema pushes you toward the bottom of the range, not the top. The grounded band runs from dbt's 72.7% (unnormalized plus semantic layer) up to 98.2% to 100% (modeled plus semantic layer), with Snowflake's roughly 90%-or-higher sitting comfortably inside it.

In the natural-language analytics builds we ship, the honest conversation with a stakeholder is always a range tied to the state of the data model, never "the benchmark says 86%, so we will get 86%." Promising a leaderboard number on a raw schema is how a project loses trust in its first demo, because the first genuinely ambiguous question exposes the gap.

Even at 90% accuracy, one in ten queries is wrong, so human review and a growing library of verified example queries are not optional, they are the mechanism that makes the number trustworthy. A wrong query that runs silently does not throw an error. It returns a confident, incorrect number that a decision-maker will act on, which is worse than no answer at all.

Two practices contain this. First, show the generated SQL before it executes and let the user confirm or correct it, and require human-in-the-loop approval before an agent acts on anything that writes or triggers a downstream decision. Recall that interaction alone nearly doubled GPT-5's BIRD-Interact score from 8.67% to 17.00%, so a confirmation step is an accuracy feature, not just a safety one. Second, capture confirmed queries as verified examples. This is exactly what Snowflake's semantic views do: verified query examples are part of what pushes Cortex Analyst to roughly 90% or higher. Every corrected query is a new grounding example, so the system compounds toward reliability instead of repeating the same mistake.

The result is a flywheel. The semantic layer sets the floor, human confirmation catches the misses, and each confirmed query becomes verified grounding that raises the floor again. Skip the loop and you have a system that is right 90% of the time and never learns which 10% it gets wrong.

Buy a grounded vendor tool if your data already lives in one platform, and build a semantic layer if your data is spread across systems or you need portable business logic. Either way, the non-negotiable is the semantic layer, not the model. That is the bottom line, and it is opinionated on purpose.

Use a grounded vendor tool when the platform and the tool line up: if you are on Snowflake, Cortex Analyst's roughly 90%-or-higher accuracy and close-to-2x edge over single-shot generation is difficult to beat by hand, so building your own is rarely worth it. Build a semantic layer when your data spans multiple warehouses or you need business logic that stays consistent across BI, notebooks, and applications, which is where a dbt-style layer earns its keep and reaches 98%-plus after modeling. Skip exactly one thing: raw schema-only text-to-SQL in production. That path is the 30-to-65% band, and it is the choice that generates the "the AI got the number wrong" incident. This is a specific instance of the broader build-versus-buy calculus for AI, and it is one of the more clear-cut cases, because the accuracy delta is enormous and well-measured.

If you take one action from this, model your metrics before you evaluate any model. Add three dbt models, or turn on semantic views, and rerun your own test set against your own ambiguous questions. That single step is what moves accuracy from a coin flip to something you can put your name on, and it is the work we do inside Particula Tech's data-and-AI engagements, benchmarking natural-language-to-SQL against your real schema before anyone promises a number. For the strategic frame around these decisions, our AI for business guide maps where text-to-SQL fits alongside the rest of an analytics stack. The model is not your bottleneck. The semantic layer is your leverage.