Checking Numbers in Table Answers with the Retriever's Own Token Alignments

When a language model answers a question from a table, the most damaging errors are small, specific ones. It reports revenue as 102.3 when the cell says 100.0, or it reads the right number from the wrong year's column. Current hallucination checkers handle these poorly. The strongest one in this file, a distilled 8-billion-parameter claim verifier for financial filings, catches only 71.4% of fabricated numbers that lie within ±5% of the truth (FinGround). The more thorough alternative, re-deriving the answer with generated code, adds several model calls and interpreter rounds to every query (PoTable; DATER).

This note proposes a cheaper route to a more exact check. The idea rests on retrievers that use late interaction, meaning they store one vector per token and score a query by matching each query token to its most similar stored token (ColBERT). If such a retriever has already indexed a table cell by cell, it holds a ready-made pointer from words to cells. We re-run each numeric claim in the generated answer as a query against the retrieved table's existing token vectors. The claim's non-numeric words, such as "net income" and "2021", locate one row header and one column header. A deterministic comparison then tests the claimed number against the value in that cell. The core claim is that binding a number to a cell through the retriever's alignments catches near-miss and wrong-cell hallucinations better than a learned verifier. It should also cost only one short query encoding and a similarity lookup per claim, with no verifier model call.

Core Claim and Mechanism

Prior work treats retrieval and verification as separate stages. LI-RAGE uses ColBERT to retrieve whole tables for open-domain table question answering. It reduces distractor hallucinations by having the reader emit a yes/no token saying whether the retrieved table is the right one, but it never uses token-level alignments to check individual answer values. The authors also leave aggregation questions to future work. FinGround does check claims at the cell level, preserving row-column coordinates when chunking, but its verification step is a distilled classifier sitting on top of a hybrid retriever that is routed by hand-built rules. We use one index for both jobs.

The mechanism works as follows. Each indexed cell keeps its coordinates $(r, k)$, and the index records which token vectors belong to row header $r$ and column header $k$. A claim $c$ is a short span with one numeric value $v$ and a set $Q$ of non-numeric query tokens. For every cell we compute a header-bound binding score:

$$s(r,k) = \sum_{i \in Q} \max_{j \in H_r \cup H_k} E_{q_i} \cdot E_{d_j}^{T},$$

Here $E_{q_i}$ is the embedding of claim token $i$, and $E_{d_j}$ ranges over the stored vectors of the tokens in row header $H_r$ and column header $H_k$. This is ColBERT's MaxSim operator restricted to the headers of a single cell, instead of the unconstrained bag of tokens that standard multi-vector retrievers use. The file notes that these retrievers ignore two-dimensional structure (WARP).

The checker takes the top-scoring cell $(r^*, k^*)$ with value $x$. It certifies the claim only if $|v - x| \le \tau(x)$, where $\tau$ reflects the rounding precision shown in the cell. It also requires that the margin between the best and second-best binding scores exceed a calibrated threshold $\lambda$. Otherwise it flags the claim or abstains.

Some claims are computed rather than looked up, for example percentage change or a sum across rows. For these, the checker takes the top-$m$ bound cells and enumerates a small fixed set of operations: difference, ratio, percentage change, and sum or mean over aligned rows. A claim counts as supported if one program matches within tolerance.

Enumeration carries a real risk. Every extra candidate value raises the chance that a fabricated number happens to match something. We will therefore measure false certification as a function of $m$ and the size of the operation set, not assume it away. The threshold $\lambda$ is chosen by split conformal calibration, following the use of conformal sets to bound retrieval-augmented errors in TRAC. TRAC worked on unstructured text, where similarity metrics are blind to small numeric differences. On a held-out calibration set of labeled claims, we pick $\lambda$ so that the rate at which hallucinated claims get certified stays below a user-set $\alpha$. This bound holds marginally, under the assumption that calibration and test claims are exchangeable. It does not hold per claim type.

Research Program

The first component is a cell-aware index. Tables are serialized cell by cell, each cell is prefixed with its row and column headers, and token vectors are tagged with coordinates. All of this sits in a late-interaction index served by WARP or PLAID, so that retrieval latency stays in the range those engines already reach.

The second component is alignment supervision. FeTaQA annotates the highlighted cells that support each reference answer. We fine-tune the retriever with a contrastive loss that pushes the binding score of a highlighted cell above that of non-highlighted cells. This also yields a direct test of whether off-the-shelf ColBERT alignments already localize cells.

The third component is the checker: a claim splitter, the binding score, exact comparison, the bounded program enumerator, and conformal calibration. Splitting answers into atomic claims follows FinGround's numerical, comparative, and computational types. We restrict ourselves to these types and do not attempt regulatory or temporal-narrative claims.

The fourth component is an analysis of where the approach stops working. We expect it to fail on claims whose wording shares no tokens with the headers ("the company's best year"), on tables with multi-level headers, and on computations outside the enumerated set. All three will be reported as abstention or error rates by category.

Baselines

The naive reference point is no verification. We measure the raw rate of wrong numeric claims in generator outputs on the same questions, which tells us how much room any checker has.

The second baseline matches a number anywhere in the table: a claim is certified if its value appears in any cell of the retrieved table, within the same tolerance. This is the ablation that isolates our key ingredient. It costs nothing and should catch near-miss values about as well as we do. It is blind to row and column swaps. If header binding adds nothing over it, the method has no reason to exist.

The third baseline uses the generator's own cited cells. The generator is prompted to output cell coordinates next to each number, and the checker compares the number against the cited cell.

The fourth baseline is a plain prompted check, in which an LLM is asked whether the claim is supported by the table. The file's evidence suggests this kind of check is weak on complex statements (TabFact) and that it reaffirms the model's own errors (Agrawal et al.).

The strongest competitor is a FinGround-style distilled verifier, an 8B classifier that decides whether each atomic claim is supported. FinGround reports 91.4% detection F1 on FinHalu, 96.2% of its GPT-4o teacher, along with 71.4% recall on near-miss values within ±5% (FinGround). The file states an 18x per-claim latency gain over the teacher but gives no absolute latency, so we will measure latency ourselves. If FinHalu and FinGround's checkpoint can be obtained, we evaluate on that data directly. If not, we train an 8B verifier on the same claim-labeled training split our calibration draws from, with the same tuning budget we spend on fine-tuning the retriever.

The expensive-accuracy reference is recomputation in the PoTable style: generate code over the table, run it, and compare the result with the claim.

All systems receive identical generator outputs and the same retrieved table, following FinGround's retrieval-equalized protocol, and they run on the same hardware. A second ablation compares frozen ColBERT alignments with alignments fine-tuned on FeTaQA, and a third compares header-bound MaxSim with unconstrained MaxSim over all table tokens.

We will count the strong baseline as beaten if the binding checker achieves higher near-miss recall and higher wrong-cell recall than the distilled verifier at a matched false-flag rate on correct claims, while running at lower measured per-claim latency. If only latency improves, that is a weaker but still useful result. If accuracy improves only on lookup claims, the claim narrows to those.

Why not just…?

Why not just string-match the claimed number against every cell? For a pure near-miss, where 102.3 is reported and no cell contains 102.3, this works and costs nothing. It is a real threat, which is why it is the central ablation. It fails when the fabricated number is real but taken from the wrong place, such as another year's column or another entity's row. That is the entity-attribute swap documented for structured records (Milecki). Our perturbation suite reports results separately for value perturbations and coordinate swaps. If swaps turn out to be rare in real generator errors, we will say that string matching is sufficient.

Why not just have the generator cite its cells and check those? Checking a cited cell is exact, but the citation comes from the same model that hallucinated the number, and fabricated grounding is a documented failure (Wang et al.). A generator that reads the wrong column will usually cite that wrong column too, so the check passes. Our binding is computed independently of the generator. The experiment that settles this is coordinate-swap recall under the two methods. We will also report a combined method that certifies a claim only when the cited cell and the bound cell agree.

Why not just use a distilled verifier or recompute with code? The verifier is the incumbent, and its own authors identify near-miss recall and the drop on outputs from non-GPT generators as its weak points (FinGround). A deterministic comparison has no dependence on which model wrote the text, and we will test this by varying the generator. Recomputation in the style of PoTable or TART handles arbitrary computations our enumerator cannot express. We expect it to win on accuracy for complex claims, and we will report the latency it pays for that. The plausible outcome is a division of labor: our checker handles lookups and simple arithmetic, and code recomputation is invoked only where our checker abstains. We will measure the fraction of claims that need the escalation.

Proposed Experiments

The first decisive check concerns localization alone. On FeTaQA, does the top binding cell coincide with an annotated highlighted cell, comparing frozen and fine-tuned ColBERT and comparing header-bound with unconstrained scoring? FeTaQA reports that a pipeline model predicted an average of 3 cells against a median of 6 highlighted cells (FeTaQA), which gives a sense of how sparse the target is. If localization is poor even after fine-tuning, the method stops here.

The second check is a controlled hallucination suite. Correct answers from FeTaQA, WikiTableQuestions, and FinQA are perturbed by ±1–5% in value, by swapping the row, and by swapping the column. Each checker is scored on recall for each perturbation type and on false flags for unperturbed claims. The suite uses FinHalu as well if it is available. No numbers are reported on Lacuna for TAT-QA, so results there will be new.

The third check uses real generator outputs, on table QA and on TabFact-style statements from several generator families, and measures how much the end-to-end rate of wrong numeric claims falls after flagging and regeneration.

The fourth check tests the conformal bound. We vary $\alpha$, check that the empirical rate of false certification on test claims tracks it, and report how the abstention rate grows as $\alpha$ tightens.

The fifth check is latency. We measure per-claim verification time against the distilled verifier and code recomputation on identical hardware, and report it alongside retrieval time for the same index. WARP reports 171ms end-to-end retrieval on LoTTE Pooled and identifies query encoding as the remaining bottleneck (WARP). For our checker the cost is likewise dominated by encoding short claims, so verification should add little beyond a retrieval call.