Teaching Financial Models Where Numbers Live: Narrative-to-Cell Pre-Training Against Extraction Hallucination

When a language model answers a question about a financial report, the most common failure is not bad arithmetic. It is reading the wrong number. The model may take the 2018 column instead of 2019, the gross figure instead of the net one, or thousands instead of millions, and then compute flawlessly on that wrong number. The best current systems already send arithmetic to a Python interpreter. Even so, TAT-LLM attributes about 48% of its remaining errors to extracting the wrong initial data, and TAGOP in TAT-QA attributes 55% to tagging the wrong cells or spans. This first-step mistake is a kind of hallucination, since the output looks well grounded but is factually wrong. It also cascades, because every later step inherits it.

We propose to attack that first step with a domain pre-training objective that financial filings supply for free. Annual reports restate their own tables in prose. A sentence such as "revenue increased 12% to $4.2 billion" in the management discussion points at specific cells in a table a few pages away. We would continue pre-training a model to point from each such sentence to the exact table cell it describes, identified by its row and column headers and its scale. We would then test whether this alignment skill, rather than domain text or generic table exposure, reduces wrong-cell extraction and its cascade on FinQA, TAT-QA and TAT-DQA. The same learned alignment also gives a check at answer time. A candidate answer can be rejected when its operands do not trace back to cells whose headers match the question. That is the kind of "well-grounded but wrong" error that lexical-overlap verifiers cannot see.

Core Claim

Domain pre-training has so far failed on numerical financial reasoning because it trained on the wrong signal. FinQA found that FinBERT, pre-trained on financial text, gave no substantial advantage over general RoBERTa on multi-step calculations. Its authors also report that about 15% of errors come from the retriever failing to find the right facts.

TAPAS at SemEval-2021 Task 9 showed the opposite side of the picture. Intermediate pre-training on tables (synthetic, counterfactual and TabFact statements) added 11 F1 points over masked language modeling alone (68.03 vs 57.01). That table corpus was generic and not tied to any domain, and the model still struggled with aggregation and unique counts.

Our claim sits between these two results. The useful domain signal is the mapping from domain language to table coordinates: which header phrase a writer means, which period, which scale. Filings contain that mapping at scale without human labels. A model continued-pre-trained on this mapping should make fewer extraction errors than one given the same number of tokens of domain text, generic table statements, or raw serialized domain tables. The error reduction should show up specifically in the extraction stage, where the cascade starts.

Research Program

Mining alignment pairs. We would collect public SEC filings of the kind PHANTOM uses (10-K and 10-Q), parse their tables, and find narrative sentences that mention a number. A sentence–cell pair is accepted only if two conditions hold. First, the number must match the cell after normalizing for scale and rounding. Second, the words around the number must overlap the cell's row header, column header, or period.

Numbers that match several cells are the label-noise problem that S3HQA addressed with teacher pseudo-labeling in retrieval. We adopt the same remedy here. A first model trained on unambiguous pairs re-labels the ambiguous ones, and pairs that stay ambiguous are dropped.

We also construct derived pairs, where the sentence states a change or ratio ("up 12%") that equals an arithmetic function of two cells. These pairs are labeled with both operand cells and the operator. That supervision is a small program, of the kind TAT-LLM learns only from annotated data.

We would exclude every company-year that appears in the FinQA, TAT-QA or TAT-DQA evaluation splits, and publish the overlap audit.

Pre-training objective. The table is serialized with explicit coordinate tokens. The model sees the sentence with its number masked and must name the source cell or cells and, for derived pairs, the operator. Writing $s$ for the masked sentence, $T$ for the serialized table, and $c^*$ for the gold cell tuple (row header, column header, scale), the alignment loss is

$$ \mathcal{L}_{\text{align}} = -\log p_\theta(c^* \mid s, T), $$

where $p_\theta$ is the model's probability over cell tuples expressed as coordinate tokens. This loss is mixed with ordinary next-token prediction on the filings, so the model keeps its fluency in domain language. The mixing ratio is one of the things we sweep.

Downstream use. After pre-training, we apply the unchanged TAT-LLM recipe: the Extractor → Reasoner → Executor pipeline, LoRA fine-tuning on LLaMA 2 7B, and a Python executor. We make one change. The Extractor emits cell coordinates, and values are looked up from the table deterministically rather than copied as text. This removes fabricated numbers entirely, so any remaining extraction error is a wrong choice of cell. That is exactly the error our objective targets.

Structural grounding check. At test time we sample several candidate programs, as in the voting setups of Beyond Confidence. Each candidate is scored by the alignment model's probability that each operand cell is the one the question's phrases refer to. The scores are then combined with vote counts.

This differs from that paper's Retrieval-Grounded Voting in a specific way. Retrieval-Grounded Voting rewards lexical overlap between the answer and the retrieved text. Wrong-cell answers have near-perfect lexical overlap, because the wrong number really does appear in the table. That paper names this "well-grounded but wrong" gap (about 4% of cases in search agents), and we expect the gap to be far larger in tables.

Baselines

The naive reference point is the TAT-LLM fine-tuning recipe applied to base LLaMA 2 7B with no continued pre-training. Its reported numbers are the anchor. TAT-LLM reports 64.60 EM on FinQA, 74.56 EM on TAT-QA and 69.45 EM on TAT-DQA for the 7B model, and 76.81 EM on FinQA for the 70B model. The same paper reports GPT-4 at 63.91 EM on FinQA and 71.92 EM on TAT-QA. We reproduce the 7B recipe ourselves, and all comparisons use our reproduction, with the published figures shown as a sanity check.

The strongest existing competitor is TAT-LLM itself, and it is the right one to beat. It already separates computation from extraction, it outperforms GPT-4 on these benchmarks, and its authors give the per-stage error attribution we aim to reduce. Winning means improving EM on all three benchmarks at the same 7B scale. More importantly, it means showing that the improvement comes from a lower share of extraction errors, measured against gold evidence, and not from the reasoner.

Three matched-compute ablations isolate the key ingredient. Each receives the same number of continued pre-training tokens and the same downstream tuning budget and seeds.

The first is continued pre-training on the text of the same filings with tables removed. This is the FinBERT-style condition that FinQA found unhelpful.

The second is TAPAS-style intermediate pre-training on generic synthetic and counterfactual table statements, following the SemEval-2021 approach.

The third is plain next-token pre-training on the same filings with their tables serialized in place but no alignment loss. This is the most important control. If domain tables seen in context are enough, the explicit pointing objective is unnecessary.

A fourth control applies the coordinate-output Extractor to the base model, so that the gain from deterministic lookup is separated from the gain from pre-training.

For the answer-time check, the baselines are majority voting, token-confidence voting, and Retrieval-Grounded Voting, all run over the same sampled candidates. Beating them means higher EM at an equal number of candidates.

Why not just…?

Why not just fine-tune on more labeled financial QA? TAT-LLM is already this approach, and 48% of its errors remain extraction errors. Supervised QA data is small and costly to annotate, while narrative-to-cell pairs are nearly free. The fair version of this objection is a data-matched comparison: does adding our mined pairs as extra fine-tuning data, with no pre-training stage, match the pre-trained model? We run exactly that condition. If it matches, the claim shrinks to "mined alignment data helps," which is still useful, and we would report it that way.

Why not just put the whole filing in a long-context frontier model? The evidence argues against it. MMQA finds a sharp performance drop past about 800 table rows. PHANTOM finds that hallucination detection degrades as context grows toward 30,000 tokens, with "lost in the middle" effects. GPT-4 already trails the 7B TAT-LLM on all three benchmarks. We still include GPT-4-class full-document prompting on TAT-DQA, where the whole document is the natural input, so this is tested rather than assumed.

Why not just expose the model to domain tables without a special objective? This is the most serious threat, because FinQA's negative result might reflect missing tables rather than a missing objective. The third ablation, plain next-token training on filings with tables serialized in place, settles it. If that condition closes most of the gap, the pointing objective is not needed and we would redirect the work toward corpus construction.

Why not just constrain the extractor to copy values from the table? Deterministic lookup removes invented numbers but not wrong-cell choices, which is where the extraction errors in these benchmarks sit. We adopt lookup in every condition and also measure it alone, so its contribution is separated from ours.

Proposed Experiments

The main experiment trains the five pre-training conditions at 7B under matched tokens, then fine-tunes each with the identical TAT-LLM recipe. Each condition is evaluated on FinQA, TAT-QA and TAT-DQA using EM.

The core diagnostic is cell-level extraction accuracy against the gold evidence annotations, both TAT-QA's derivation evidence and FinQA's supporting facts. We also re-run the TAT-LLM-style error attribution for every condition. The central prediction is that the share of extraction errors falls clearly below 48%, while reasoning-stage errors stay roughly flat. If EM rises but the extraction share does not fall, the mechanism we propose is wrong.

TAT-DQA tests the retrieval-cascade setting, since it has no predefined table boundaries. There we also measure whether the page and region selected upstream contain the gold cells, and we separate cascade failures from within-table selection failures.

Two further checks bound the claim. The first is an out-of-domain control on SemEval-2021 Task 9 scientific tables, with the TAPAS figure of 68.03 micro F1 as reference. There we predict little or no gain over generic table pre-training. Gains that appear only in the financial domain would support the claim that the alignment is domain language grounded in domain tables, not general table skill.

The second is the answer-time comparison. We compare the structural grounding check against confidence voting and Retrieval-Grounded Voting at 4 and 8 candidates, and report accuracy specifically on questions where the majority answer uses correct numbers from the wrong cells.

Finally, a sweep over the mixing ratio of the alignment loss and next-token loss, together with a check on the FinQA test split for contaminated company-years, guards against the gains coming from memorized filings rather than learned alignment.