TL;DR
Successful SQL execution doesn't guarantee a query is actually doing what the business wants, especially as definitions for metrics like "revenue" or "refunds" evolve. This paper introduces an executable verification profile that uses targeted synthetic data (legal counterexamples) to catch semantic errors that current database snapshots miss, outperforming generic testing and LLM-based "judges."
What Problem Does This Paper Address?
In the world of AI-driven analytics, we often ask models to generate SQL queries from natural language. While we have tools to check if a query is syntactically valid or if it runs on a current database, we lack robust ways to ensure semantic correctness—does the query actually implement the specific business logic requested?
The authors highlight a phenomenon called "accidental agreement." Imagine a business rule changes how refunds are attributed to months. On your current database, which happens to have no refunds for January, an old query and a new query might return the exact same number. The query looks "correct" today, but it will fail the moment a refund is recorded. As business definitions for taxes, exchange rates, and eligibility criteria drift, existing "fixed" queries can silently become invalid.
Methods
The authors propose a framework for business-semantic revalidation. Instead of relying on a single database snapshot, they generate a suite of "legal" synthetic databases (ledgers) designed to expose discrepancies.
Key components include:
- Correctness Contract: A formal binding of a query to a specific business revision, report interval, and "knowledge cutoff" (what the system knew at a specific time).
- Legal Data Construction: A sampler that generates synthetic data obeying strict business constraints (e.g., you can't refund more than the original sale amount).
- Obligation-Guided Checking: This is the core innovation. The system identifies "obligations"—specific data scenarios where two different business definitions should produce different results (e.g., a sale with a late-recorded refund). It then prioritizes testing on ledgers that cover these obligations.
- Independent Evaluator: A reference implementation of the business logic (written in Python using high-precision decimal math) acts as the "ground truth" against which the SQL query's output is compared.
The workflow is designed to be auditable, as shown in the process diagram:

Datasets
The study avoids the "easy" path of using standard benchmarks like Spider, which often lack complex temporal business logic. Instead, they introduced:
- 32 Newly Authored Revision Families: These cover realistic enterprise scenarios like changes to tax treatment, refund attribution, and exchange-rate dates.
- 192 Query-Definition Cells: Used for a preregistered confirmatory study.
- Natural-Query Pilot: They used LLMs (specifically
gpt-oss-20bandgemma-4-12b) to generate SQL for these families to see how "natural" errors manifest in the wild.
Key Insights and Ideas
- The "Current Snapshot" Trap: In their confirmatory study, 63 out of 144 nonconforming queries actually agreed with the reference implementation on the current database snapshot. If you only test against your production data, you'll miss nearly half of the logic errors.
- Coverage over Priority: The authors tested whether weighting "new" rules more heavily helped find errors faster. Surprisingly, they found that simply ensuring coverage of all business logic branches (obligations) was the real driver of performance, regardless of how those branches were weighted.
- LLMs are Poor Judges: When they asked LLMs to "judge" if a query was correct by looking at the code and definitions, the models were unreliable—either hallucinating errors in correct queries (high false rejection) or missing subtle logic flaws. Execution-based verification with a "replayable witness" (a specific data instance that proves the error) is far superior.
Key Results
The "Obligation-Guided" approach significantly outperformed generic random testing:
- Detection Rate: At an eight-ledger testing budget, the guided method detected 137 out of 144 discrepancies, compared to only 127 for generic ordering.
- Statistical Significance: This represents a 6.9 percentage point gain (p = 0.0156), proving that targeted counterexamples are more efficient than random ones.
- Soundness: Every ledger-based rejection was backed by a "legal witness"—a specific data state where the query failed—resulting in zero false rejections for the automated checker.
The claims and evidence are summarized in the paper's first table:
Limitations and Future Work
- Scope: The study was conducted within an authored schema and local-model scope. While the logic is generalizable, the specific "obligations" were hand-designed for this study.
- Unresolved Cases: 12 out of 192 cells remained "unresolved," where the system couldn't definitively prove or disprove conformance within the budget.
- Future Extensions: The authors have planned a "Prospective External Extension" to test this framework against independent, large-scale enterprise schemas like BIRD or Spider 2.0 to see how well the "obligation" generation scales to massive, unfamiliar databases.
Summary updated