Checking Claims Against the Data That Existed When They Were Made

Many viral claims are about official numbers, such as unemployment, crime counts, hospital admissions, or budget lines. The numbers behind them live in databases that change over time. Statistics get revised, late records arrive, and corrections overwrite earlier values. A politician who said "unemployment fell to 3.9% in March" may have quoted the figure published that week exactly, even though today's database shows 4.1% for March. Today's automated fact-checkers look up the current value and call the statement false. A human fact-checker would say it was accurate when spoken and has since been revised. Nobody currently builds systems that make this distinction, and nobody measures whether they can.

We propose as-of verification. The system translates a claim into a database query that has two separate time settings. The first is the period the claim is about. The second is the moment the claim was made, which fixes which version of the data the speaker could have seen. The system runs the query against a database that keeps its revision history. It then returns one of three kinds of verdict: true as of the claim, false as of the claim, or true as of the claim but contradicted by later revisions. The core bet is that one piece is the missing, testable part: working out which version of the data a claim refers to, without being told. Text-to-SQL, meaning the translation of plain-language questions into database queries, is largely a solved component in comparison. We also expect the three-way verdict to change a large share of labels on real statistical claims.

Where this sits

Two systems already check claims against structured data. Both ignore revision history. AggChecker searches for the SQL query that best explains a numeric claim and compares that query's result to the stated value. It reaches a macro F1 of 47.9% in fully automated mode on 392 real claims. It treats the database as a single timeless snapshot, and its query space has no time windows or inequalities. Mori et al. translate economic claims into temporal logic queries over one-dimensional time series with a fine-tuned T5 model. They reach 0.90 accuracy, but only on 5,000 synthetic claims built from 10 templates, and they also check only against current data. We reuse AggChecker's idea of verifying by executing a candidate query. We reuse Mori et al.'s idea of making time explicit in the query. We add the second time axis, and we evaluate on real claims.

Temporal fact-checking over web text motivates the problem. Allein et al. (2020) show that restricting or ranking evidence by the claim's date improves veracity F1 on the MultiFC dataset of real fact-checked claims. Allein et al. (2023) show that encoding both publication-date offsets and time expressions inside the text lifts a BiLSTM from 32.39% to 47.43% macro F1. Both treat time as a soft signal on text snippets and never compute anything. QuanTemp and QuanTemp++ add strict date cutoffs to stop evidence published after the claim from leaking into retrieval. That is the web-search analogue of our "as of claim time" setting. QuanTemp++ also reports that better retrieval (NDCG@10 rising from 0.33 to 0.57) buys only about 3.6% macro F1 in verification. We read that result as a constraint on design: the verifier must compute the number, not read about it.

Text-to-SQL work both supplies the method and limits what we can expect from it. EHRSQL shows that 93.2% of real clinical questions involve time columns. It also shows that a general-domain parser transfers at 4.7% execution accuracy. Joy et al. show a single-pass LLM scoring -713.37 on the penalized reliability metric over MIMIC-IV, which means wrong queries are common and costly. Redd et al. show that an agent loop that inspects rows, executes queries, and reads the errors raises accuracy from 28.6% to 91.4%, though on only 35 queries. TempAgent shows that filtering by time inside retrieval tools beats leaving time to the LLM's context, reaching 70.2% Hits@1 on the MultiTQ benchmark. We take the agent loop and the time-filtering tools as given components.

Core Claim

Let $c$ be a claim made at time $t_c$ that asserts a value $y$ about a period $V$. Let $D^{(\tau)}$ be the database as it stood at time $\tau$ (its transaction time). Let $q$ be the query the claim implies. The as-of verdict compares

$$v_{\text{then}} = q\big(D^{(t_c)}, V\big), \qquad v_{\text{now}} = q\big(D^{(\text{now})}, V\big)$$

against $y$ using a tolerance $\epsilon$ that the system infers from the claim's own precision. For example, "about 4%" implies a wider tolerance than "3.9%". The claim is true-as-made if $|v_{\text{then}} - y| \le \epsilon$. It is flagged as since-revised if, in addition, $|v_{\text{now}} - y| > \epsilon$.

The hard inferences are $t_c$, $V$, and $\epsilon$. Claims rarely state them, and the obvious default of using current data is wrong exactly on the claims that matter most for fairness to speakers. Our claim is that an agent which infers these three quantities explicitly will be accurate on "revision-flip" claims, meaning claims whose verdict differs between $v_{\text{then}}$ and $v_{\text{now}}$. It will also give up little on ordinary claims. A strong text-to-SQL agent that is simply handed the schema should fail on these flip claims even when the revision history is available to it.

Research Program

The first product is a benchmark, which we call VintageCheck, with three parts of increasing realism.

The first part converts EHRSQL questions into claims. Each answer is either kept or perturbed, so the claim is true or false. The database is made bitemporal by injecting late-arriving and corrected records with commit timestamps. This gives a controlled, large testbed where flip rates can be set on purpose.

The second part extends the FRED claim templates to economic series whose published revision history we store as dated versions. This gives realistic revision patterns while claims stay synthetic.

The third part is the real-world test. We take statistical and temporal claims from QuanTemp and the AggChecker corpus that cite an identifiable official series. We link each claim to that series and its revision history, and annotators label both the as-made and current verdicts. We expect this subset to be small, in the hundreds of claims, and it is the one that decides whether the idea matters.

The second product is the system. It is an agent loop in the style of Redd et al. with three additions. The first is a grounding step that proposes $t_c$ from claim metadata and $V$ from time expressions in the text. It falls back to candidate sets when unsure, rather than relying on a fixed rule-based time tagger, since Allein et al. (2023) document how brittle such taggers are. The second is version-selection tools that restrict any query to $D^{(\tau)}$, analogous to TempAgent's Before, After, and On filters. The third is a symbolic comparison step that computes the verdict from executed values rather than asking the LLM to judge. When no candidate query both executes and matches the claim's measured quantity, the system abstains, using EHRSQL-style entropy thresholding. The goal is to avoid the forced-wrong-query failure seen in Joy et al..

We treat scalability as a measured constraint rather than a separate contribution. We report LLM calls, SQL executions, and wall-clock time per verified claim. We also measure how often claims about the same series and period can share cached query results, following AggChecker's use of caching.

Baselines

The naive reference points are a claim-only classifier and a "current snapshot" check. The claim-only classifier bounds how much can be guessed from wording alone: TabFact reports about 50% for its blind baseline (TabFact), and MultiFC reports 25.3% macro F1 for claim-only (Augenstein et al.). The current-snapshot check runs our full pipeline with $\tau$ fixed to now. On revision-flip claims it is wrong by construction, so it serves as a floor that shows how often the problem arises, not as a competitor.

The strongest existing method is the agentic text-to-SQL loop of Redd et al., turned into a verifier. It generates and repairs SQL until the query executes, then compares the result with the claimed value. It reports 91.4% query accuracy on its 35-query benchmark. We have found no reported result on Lacuna for it as a claim verifier. To make the comparison fair, it gets the same backbone LLM, the same bitemporal schema including commit-timestamp columns, the same budget of calls per claim, and a prompt that states the claim's date. What we withhold is only our explicit grounding step and version tools. Beating it means higher accuracy on flip claims in the real-world subset, with a 95% bootstrap interval that excludes zero, and no loss beyond one point on non-flip claims.

For comparison with published systems, we run AggChecker on its own corpus, where it reports 47.9% fully automated F1. We run T5 in the style of Mori et al. on the FRED portion, where it reports 0.90 accuracy on current data. On the QuanTemp-derived claims we run the best web-evidence verifier from QuanTemp, which reports 58.32 macro F1, under its date-cutoff protocol.

Three ablations isolate the key ingredient. The first replaces inferred $t_c$ with the true claim date, which gives an oracle upper bound. The second removes the tolerance inference. The third removes the symbolic comparison and lets the LLM read the executed values and judge.

Why not just…?

Why not just give a strong text-to-SQL agent the bitemporal schema and the claim date? This is the real threat. If the agent reliably adds the right version filter on its own, our grounding step adds nothing, and the contribution shrinks to the benchmark and the measured flip rate. The fair Redd-style baseline described above tests exactly this. Our prediction is that it succeeds when claims name their period explicitly. We expect it to fail when the period is implicit ("last month's jobs report") or when the claim quotes a preliminary estimate. If the gap on the real-world subset falls within noise, we will report that and present VintageCheck as the contribution.

Why not just retrieve web evidence with temporal ranking? Date-aware web retrieval (Allein et al., 2020; QuanTemp++) can surface the figure published at claim time. However, QuanTemp++ shows that better retrieval barely moves verification, and QuanTemp reports LLMs misreading numbers in the snippets they retrieve. A web pipeline also cannot compute derived quantities, such as a change between two administrations, that no article states. We will measure its accuracy on derived versus directly quoted claims separately.

Why not just put the relevant tables in the LLM's context? Table-BERT falls from about 79% to about 58% on multi-row claims in TabFact. EHRSQL tables exceed 300 million rows, and revision histories multiply that size. We include a context-stuffing baseline only on the small FRED series, where it is feasible. There it tests whether in-context reading can handle revision comparison at all.

Proposed Experiments

The decisive check is accuracy on revision-flip claims in the real-world subset, compared with the fair agentic baseline. We report flip-claim accuracy, macro F1 over the three verdicts, abstention precision, and cost per claim.

A second check establishes that the problem matters. We will report the fraction of QuanTemp-derived statistical claims whose verdict flips between as-made and current data. If that fraction is tiny, the three-way verdict is a curiosity.

On the EHRSQL-derived part, we vary the injected revision rate. We predict that the gap between our system and the current-snapshot baseline grows roughly linearly with the flip rate, since non-flip claims should be handled equally. We also predict that the gap to the oracle-$t_c$ ablation stays constant, which would locate the remaining error in time inference.