Verifying Text-to-SQL Queries under Changing Business Semantics

Genliang Zhu, Xiaobo Chen, Ziyuan Wang, Chu WangarXiv 2026 · arXiv

TL;DR

Successful SQL execution doesn't guarantee a query is actually doing what the business wants, especially as definitions for metrics like "revenue" or "refunds" evolve. This paper introduces an executable verification profile that uses targeted synthetic data (legal counterexamples) to catch semantic errors that current database snapshots miss, outperforming generic testing and LLM-based "judges."

What Problem Does This Paper Address?

In the world of AI-driven analytics, we often ask models to generate SQL queries from natural language. While we have tools to check if a query is syntactically valid or if it runs on a current database, we lack robust ways to ensure semantic correctness—does the query actually implement the specific business logic requested?

The authors highlight a phenomenon called "accidental agreement." Imagine a business rule changes how refunds are attributed to months. On your current database, which happens to have no refunds for January, an old query and a new query might return the exact same number. The query looks "correct" today, but it will fail the moment a refund is recorded. As business definitions for taxes, exchange rates, and eligibility criteria drift, existing "fixed" queries can silently become invalid.

Methods

The authors propose a framework for business-semantic revalidation. Instead of relying on a single database snapshot, they generate a suite of "legal" synthetic databases (ledgers) designed to expose discrepancies.

Key components include:

  • Correctness Contract: A formal binding of a query to a specific business revision, report interval, and "knowledge cutoff" (what the system knew at a specific time).
  • Legal Data Construction: A sampler that generates synthetic data obeying strict business constraints (e.g., you can't refund more than the original sale amount).
  • Obligation-Guided Checking: This is the core innovation. The system identifies "obligations"—specific data scenarios where two different business definitions should produce different results (e.g., a sale with a late-recorded refund). It then prioritizes testing on ledgers that cover these obligations.
  • Independent Evaluator: A reference implementation of the business logic (written in Python using high-precision decimal math) acts as the "ground truth" against which the SQL query's output is compared.

The workflow is designed to be auditable, as shown in the process diagram:

![Fig.

  1. Primary labels are frozen independently before arm execution and remain hidden. The final replay checks frozen decisions without altering those labels or guiding search.](/figures/art_3b6a84e06d584923820e61f0b7e1ba09)

Datasets

The study avoids the "easy" path of using standard benchmarks like Spider, which often lack complex temporal business logic. Instead, they introduced:

  • 32 Newly Authored Revision Families: These cover realistic enterprise scenarios like changes to tax treatment, refund attribution, and exchange-rate dates.
  • 192 Query-Definition Cells: Used for a preregistered confirmatory study.
  • Natural-Query Pilot: They used LLMs (specifically gpt-oss-20b and gemma-4-12b) to generate SQL for these families to see how "natural" errors manifest in the wild.

Key Insights and Ideas

  • The "Current Snapshot" Trap: In their confirmatory study, 63 out of 144 nonconforming queries actually agreed with the reference implementation on the current database snapshot. If you only test against your production data, you'll miss nearly half of the logic errors.
  • Coverage over Priority: The authors tested whether weighting "new" rules more heavily helped find errors faster. Surprisingly, they found that simply ensuring coverage of all business logic branches (obligations) was the real driver of performance, regardless of how those branches were weighted.
  • LLMs are Poor Judges: When they asked LLMs to "judge" if a query was correct by looking at the code and definitions, the models were unreliable—either hallucinating errors in correct queries (high false rejection) or missing subtle logic flaws. Execution-based verification with a "replayable witness" (a specific data instance that proves the error) is far superior.

Key Results

The "Obligation-Guided" approach significantly outperformed generic random testing:

  • Detection Rate: At an eight-ledger testing budget, the guided method detected 137 out of 144 discrepancies, compared to only 127 for generic ordering.
  • Statistical Significance: This represents a 6.9 percentage point gain (p = 0.0156), proving that targeted counterexamples are more efficient than random ones.
  • Soundness: Every ledger-based rejection was backed by a "legal witness"—a specific data state where the query failed—resulting in zero false rejections for the automated checker.

The claims and evidence are summarized in the paper's first table:

Table 1. Claims, executed evidence and interpretation. Qualification and future designs do not enlarge the empirical denominator.

Limitations and Future Work

  • Scope: The study was conducted within an authored schema and local-model scope. While the logic is generalizable, the specific "obligations" were hand-designed for this study.
  • Unresolved Cases: 12 out of 192 cells remained "unresolved," where the system couldn't definitively prove or disprove conformance within the budget.
  • Future Extensions: The authors have planned a "Prospective External Extension" to test this framework against independent, large-scale enterprise schemas like BIRD or Spider 2.0 to see how well the "obligation" generation scales to massive, unfamiliar databases.

Summary updated

Abstract

In text-to-SQL and AI-agent analytics, successful SQL execution does not establish semantic correctness under changing business definitions. We study revalidation of fixed queries after changes to refund attribution, tax treatment, eligible populations and exchange-rate dates. An executable verification profile binds the requested revision and knowledge cutoff to legal counterexample data and exact monetary comparison. A preregistered confirmation on 32 newly authored revision families and two local producer bundles retains 192 query-definition cells: 144 nonconforming, 36 finite-suite agreeing and 12 unresolved. Of the nonconforming cells, 63 agree on the current snapshot. At an eight-ledger budget, obligation-guided checking detects 137 discrepancies versus 127 for generic ordering, a paired difference of 6.9 percentage points (family bootstrap 95% interval [2.1, 13.6]; exact family sign-flip p=0.0156). Every ledger arm rejects zero of the 36 agreeing cells. Current-obligation and equal-weight scoring match the guided result, identifying coverage rather than revision-specific priority as the observed source of benefit. Controlled assays, a natural-query pilot and repeated samples provide separate mechanism and robustness evidence. A legal discrepancy refutes conformance; agreement remains restricted to the recorded suite. The study establishes an auditable business-semantic revalidation profile within its authored schema and local-model scope.
BibTeX
@misc{zhu2026verifying,
  author = {Genliang Zhu and Xiaobo Chen and Ziyuan Wang and Chu Wang},
  title = {{Verifying Text-to-SQL Queries under Changing Business Semantics}},
  year = {2026},
  eprint = {2610.41090},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41090},
}
Figure