Many factual claims in journalism, public health and business reporting are statements about databases: "the region with the most clinics also had the lowest vaccination rate," or "every product launched after 2020 sold more units than its predecessor." Checking such a claim means stitching together facts that live in several tables, linked through shared identifier columns called foreign keys. A natural shortcut is to turn the claim into a question, have a language model write a database query (SQL), run it, and compare the answer with the claim. This shortcut has a hidden flaw. When the query is wrong, because it follows the wrong chain of tables or applies a wrong filter, its output disagrees with the claim, and the system reports "refuted." A broken query and a false claim look the same from the outside. On the hardest public text-to-SQL benchmark, even GPT-4 with a strong prompting pipeline writes a correct query only about half the time. A verifier built this way would therefore issue many confident refutations whose cause is its own mistake.
This note proposes a verifier that splits a claim into small atomic conditions. Each condition is tied to a specific path through the tables, and the conditions are executed incrementally, so the system can say which condition fails, which table and rows cause the failure, and whether the failure survives alternative plausible routes through the schema. The output is not just a label. It is a set of database rows that witness the verdict, plus a pointer to the culprit condition. When plausible routes through the schema disagree, the system reports "schema-ambiguous" instead of "refuted." Because the claims are built from benchmarks that ship correct SQL, we can measure for the first time how often a database fact-checker's refutations are caused by its own query errors, and whether localization reduces that rate.
Core Claim
The claim is that refutations over multi-table databases can be made trustworthy by three changes, without a larger model. The first is executing the claim as a conjunction of atomic conditions, each attached to its own join path. The second is returning provenance, meaning the exact database rows that produced each intermediate value, as the explanation. The third is treating disagreement across alternative join paths as a separate outcome rather than as evidence against the claim. Together these should cut the rate of spurious refutations (refutations whose underlying query is wrong) compared with a single monolithic SQL query, at equal or better verdict accuracy. They should also make culprit localization possible at the level of table, column and cell.
Where This Sits
The closest prior system is AggChecker. It verifies aggregate claims in news articles against relational data by matching keywords to candidate SQL queries and using a document-level probabilistic model. Users checked summaries six times faster with it. AggChecker is limited to aggregate functions, depends on a human to choose among candidate queries, and gives no localization of why a claim fails. We keep its framing of claims as database queries but replace the human disambiguation step with automated cross-path agreement and atomic localization.
Single-table fact verification supplies both the motivation and the methods. TabFact established the benchmark and documented the trade-off between soft linguistic matching and brittle symbolic programs. SASP shows that programs learned from binary labels are often spurious: they get the right label with the wrong logic. Its structure-guided parser raises exact program-tree matching on Logic2Text from 15.6% to 47.9%. LERGV responds to the same brittleness by treating several candidate programs as soft graph evidence rather than executing one. Yang & Zhu decompose claims into sub-statements from four logic templates. All four papers work on one flat table.
Our design takes a position in the SASP-versus-LERGV tension. Like SASP, we keep deterministic execution, so arithmetic is exact. Like LERGV, we use several candidate programs, but only to estimate whether the schema route is pinned down, never to soften the arithmetic.
The decomposition logic comes from LOREN. LOREN marks a claim Supported only if every phrase is supported and Refuted if any phrase is refuted. It reports a 74.3 F1 at identifying the culprit phrase on text. We apply the same rule, but our atoms are SQL predicates over join paths rather than latent phrase variables, so their truth values are computed rather than predicted.
On the multi-table side, Graph-Link supplies the schema-routing method. Schema linking means deciding which tables and columns a query needs. Graph-Link frames it as a node-weighted Steiner tree, the cheapest subgraph of the schema graph that connects all tables the claim mentions. This recovers "bridge tables" the claim never names, reaching a 90.8% hit rate on BIRD dev. We go beyond Graph-Link by taking the top several Steiner trees rather than one, and by turning their agreement into a verification signal.
ProTrix motivates the gap directly. Its plan-then-reason approach mixes SQL with textual reasoning, but its authors state that it does not handle multi-table joins. FEVEROUS and PubHealthTab likewise pair each claim with an isolated table.
Research Program
The verifier has four stages, all running on an open model of the 7–8B class with a SQL engine.
The first stage decomposes the claim into atoms. A prompted or lightly fine-tuned model rewrites the claim as a small logical form: a set of entity anchors, atomic conditions (filters, comparisons, aggregates, superlatives) and the logical operator joining them. Each atom keeps a pointer to the span of claim text it came from. This keeps the "culprit phrase" notion from LOREN, and it makes explanations readable to a human checker.
The second stage routes each atom through the schema. For the tables each atom touches, we compute the top $k$ Steiner trees over the schema graph in the style of Graph-Link (we expect $k$ around 3–5). Each tree defines a join path. The model then writes one SQL fragment per atom per path. Fragments are executed incrementally along the path, so we can see where a condition first becomes empty or contradicted. Examples include a join that yields no rows, a filter on a downstream table that removes the claimed entity, or an aggregate that differs from the claimed number.
The third stage records why-provenance for each fragment. Why-provenance is the set of source rows, across all joined tables, that contributed to the fragment's result, and it is cheap to obtain by carrying primary keys through the query. For a supported atom, these rows are the witness. For a refuted atom, the explanation consists of the contradicting rows, the claim span tied to the atom, and the value the database actually holds. That value doubles as a minimal repair of the claim.
The fourth stage aggregates the atoms with the LOREN rule within each join path, and then across paths. If every plausible path agrees, the verdict is Supported or Refuted with the culprit atom attached. If the paths disagree on any atom, the output is "schema-ambiguous," together with the competing paths. This third outcome is the step that should most reduce spurious refutations, and it also addresses the negative-prediction bias that SEVA documents. In SEVA, uncertain verifiers default to the negative label. Here, uncertainty about the query has its own label.
The benchmark has to be built, because no multi-table claim-verification dataset with culprit labels appears in the file. We will turn questions from BIRD and Spider, which come with correct SQL, into declarative claims. A supported claim states the gold answer. A refuted claim perturbs a single atom: an aggregate value, a comparison direction, a filter constant, or an attribute that lives only in a joined table reached through a bridge table. Because exactly one atom is perturbed, the gold culprit is known at the level of span, table and column. Because gold SQL exists, we can check whether a system's query was correct independently of its verdict. We will balance refutations whose culprit sits in the anchor table against those whose culprit sits two or more joins away.
Templated claims are unnatural, so we will also hire annotators to write about 500 free-form claims over BIRD databases and label them. Where its relational datasets are recoverable, we will re-use AggChecker's 392 real news claims for a test of the aggregate-only setting.
Baselines
The naive reference point is a claim-only classifier that sees the claim text but not the database. It measures how much the perturbation templates leak the label. FEVEROUS reports that a claim-only baseline already reaches 0.58 accuracy there. If our claim-only number is far above 50% on balanced data, the templates must be revised before any other result is trusted.
The strongest competitor is a monolithic text-to-SQL verifier. It rewrites the claim as a question, generates one SQL query with a DIN-SQL-style decomposed prompt over a schema linked by Graph-Link, runs the query, and compares the result with the claimed value. It is the strongest baseline because it already combines the best schema routing in the file with exact execution, so it isolates exactly what our atomic, multi-path design adds. The relevant reported numbers are BIRD's 55.90% test execution accuracy for GPT-4 + DIN-SQL and Graph-Link's 90.8% schema-linking hit rate on BIRD dev. For claim verification over multi-table databases, no reported result was found on Lacuna. To keep the comparison fair, the baseline uses the same backbone model, the same Graph-Link linker, the same prompt-tuning budget on a shared development split, and the same number of total model calls, so that it can sample $k$ queries and majority-vote. Beating it means matching its verdict macro-F1 while reducing its spurious-refutation rate by a clear margin; we would treat a 30% relative reduction as meaningful. It also means producing culprit localization, which the baseline cannot provide at all.
A second baseline is a frontier model given chain-of-thought prompting. It sees the schema plus rows sampled along the gold join path, and outputs a verdict and a free-text explanation without executing anything. It represents what a practitioner would try first.
A third is AggChecker-style keyword-to-SQL matching on the aggregate subset, run as originally designed but with automatic top-1 selection in place of the human choice. Its role is to show how much the learned components add over the pre-LLM pipeline.
Three ablations isolate the key ingredients. The first keeps atomic decomposition but uses a single join path; this removes the agreement signal. The second keeps multiple paths but executes one monolithic query per path; this removes localization. The third removes provenance and returns only the culprit span; this tests whether the witness rows matter to human checkers.
Why not just…?
Why not just convert the claim to a question and use the best text-to-SQL system? This is the real threat, and it is the main baseline. Verdict accuracy alone may show no difference, because a wrong query is sometimes still compared correctly by luck. The decisive measurement is the one this baseline cannot hide from: the fraction of its Refuted verdicts whose query differs in execution from the gold SQL. With BIRD-level query accuracy near 56%, we expect this fraction to be large. If the monolithic verifier's spurious-refutation rate turns out to be comparable to ours, the contribution shrinks to culprit localization and the human-study effects, and we will report it that way.
Why not just denormalize the database into one wide table and use a single-table fact-checker? BIRD databases reach 33.4 GB with tables of up to 549K rows, so a full pre-join explodes in size. SAT reports a 5–9% accuracy loss when structure-aware masking is removed even on single tables. To give this alternative its best chance, we include an oracle variant that flattens only along the gold join path on the smaller Spider databases. If that oracle matches us, the lesson is that routing, not decomposition, is the bottleneck. The single-path ablation tests the same question.
Why not just prompt a frontier model with chain-of-thought and skip execution? ProTrix reports that purely textual reasoning is imprecise at counting, sorting and filtering, and BIRD shows GPT-4 at 34.88% without external knowledge. Free-text explanations are also exactly what drives the overconfidence documented in Generating Fact Checking Briefs, where about 45% of workers relied on briefs without checking. We run this baseline and include its explanations in the human study, so any effect of unverifiable rationales on human accuracy becomes measurable.
Proposed Experiments
The headline experiment reports four numbers per system on the BIRD-derived and Spider-derived claim sets. They are verdict macro-F1; the spurious-refutation rate, measured against gold SQL; culprit F1 at the span, table and column levels, following LOREN's culprit metric; and abstention precision, meaning the fraction of schema-ambiguous outputs where every single-path query was in fact wrong. Results are broken down by the number of joins between the anchor table and the culprit. The design predicts that our advantage grows with join depth, because bridge tables are where single-query routing fails.
A second check moves to the 500 human-written claims to test whether these gains survive natural phrasing. A third uses the AggChecker claims to test the real-world aggregate setting.
The final check is a human study with fact-checkers verifying claims under three conditions: a monolithic SQL query with its result, a chain-of-thought explanation, and our witness rows with the culprit pointer. We measure accuracy, time to verdict, and the rate at which checkers accept a system verdict that is wrong. The proposal succeeds if witness-backed verdicts lower that acceptance rate without slowing checkers down.
References
- Verifying Text Summaries of Relational Data Sets
- TabFact: A Large-scale Dataset for Table-based Fact Verification
- Learning to Generate Programs for Table Fact Verification via Structure-Aware Semantic Parsing
- Logic-level Evidence Retrieval and Graph-based Verification Network for Table-based Fact Verification
- Exploring Decomposition for Table-based Fact Verification
- LOREN: Logic-Regularized Reasoning for Interpretable Fact Verification
- Graph-Link: Bridging the Semantic-Structural Gap in Text-to-SQL via Constrained Subgraph Induction
- ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context
- FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information
- PubHealthTab: A Public Health Table-based Dataset for Evidence-based Fact Checking
- SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution
- Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
- Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
- Table Fact Verification with Structure-Aware Transformer
- Generating Fact Checking Briefs