Address, Don't Copy: Type-Aware Defenses Against Distraction in Table-and-Text Reasoning

Language models answering questions over financial reports and tables fail mostly because they grab the wrong evidence, not because they cannot do arithmetic. In TAT-QA, 55% of the errors made by the strongest hybrid model came from tagging the wrong cells or text spans (TAT-QA). When GPT-4 read a whole financial report instead of retrieved passages, its score on UDA's FinHybrid fell from 45.9 to 0.4 (UDA). We argue that this distraction has a type structure that current defenses ignore. A table mixes two kinds of information. The discrete keys are row labels, column headers, years, entity names and scale words like "in millions". The continuous values are the numbers in the cells. A language model handles both as ordinary text tokens, so a number is just a string of digits and its magnitude means nothing to the model. Distraction can therefore enter through two separate doors. The first is confusable keys: a "Total revenue" row in the wrong segment table, or the right row in the wrong year. The second is confusable values: nearby numbers of the same size that the model miscopies while writing out an equation.

The proposal has two parts. First, we measure which door matters, using distractors generated from the table structure so that the gold answer provably does not change. Second, we build a reasoner whose language model never writes a number. It emits cell addresses, meaning tuples of discrete keys together with a scale. A deterministic executor looks up the continuous values and does the arithmetic. We then train the model specifically against key-collision distractors. Put briefly, the language model keeps only the discrete part of the problem, and we harden it against the kind of noise that attacks that part.

Core Claim

We make two testable claims about hybrid table-text question answering with 7B–14B open models, the size class that Breaking Focus and Orchestra show to be most fragile under distraction.

The first claim concerns where errors come from. Under realistic noise, errors split into two groups. One group is a value-transcription component: digits copied wrongly, or the wrong scale applied. Removing numbers from the generated text eliminates this component. The other group is a key-binding component: the right kind of quantity pointed at in the wrong row, column or year. This component remains after numbers are removed, and it accounts for most of the remaining degradation.

The second claim is about the fix. A model that addresses cells instead of copying values, and that is trained with preference pairs built from key-collisions, should degrade much less under typed distraction than three alternatives. These are generic distraction-preference training, soft relevance weighting, and program generation that copies numbers.

If the diagnostic in the first claim shows that value transcription is negligible, the addressing component is unnecessary. The proposal would then reduce to typed adversarial training, and we would report it as such.

Relation to Prior Work

Three lines of work motivate the gap. The benchmarks TAT-QA and FinQA pair tables with report text and attribute large shares of errors to evidence extraction and retrieval. Neither tells us which surrounding content causes the mistake. UDA shows that numerical reasoning collapses in long documents and that perfect context helps by up to 60%. It does not analyze which cells or spans cause the collapse. Breaking Focus establishes three relevant facts: distraction is an ability failure rather than a knowledge failure, prompting the model to "ignore irrelevant context" does not work, and DPO on clean versus distracted pairs restores robustness. Its distractors are semantic and untyped, and it works on multiple-choice QA rather than tables. We reuse its training recipe but change what the preference pairs contain.

Several methods supply techniques we build on. AbstRaL shields grade-school math from instantiation and distractor shifts by mapping text to symbolic variables and solving with SymPy. Its condition-recognition step still copies the numbers (in0=12), and it never touches tables, keys or scales. Our addresses play the role of its variables but are bound to table coordinates, so a copying error cannot occur. FinQA's FinQANet and TAT-QA's TAGOP already separate evidence selection from symbolic operators, and TAGOP predicts scale as a separate discrete output. Our design keeps that division of labor but puts it inside a modern language model. We also add the typed robustness training they lack. Chain-of-Table prunes tables with discrete row and column operations, and CABINET down-weights irrelevant cells softly. CABINET also shows that hard pruning compounds errors when a gold cell is wrongly deleted. Both methods reduce how much noise the model sees, and both still treat numbers as text. Neither targets distractors that look relevant by key, which no relevance filter can remove without also removing the gold cell.

On the tabular-learning side, TabNAT and LassoFlexNet argue that discrete and continuous columns need different treatment. TabNAT uses separate categorical heads and diffusion heads. LassoFlexNet uses per-feature encodings and gating. We take that design principle, not their code: continuous values should never be forced through a mechanism built for discrete tokens.

Research Program

The first component is a typed distraction generator for TAT-QA, FinQA and WikiTQ. Every gold answer is computed from a set of cells, and each cell has a key tuple (row label, column header, table or section identity, period) and a scale. The generator inserts content of four controlled types:

  • Key-collision distractors share all but one key with a gold cell. Examples are the same row label in a different segment table, or the same metric for an adjacent year.
  • Value-proximity distractors are numbers within a small relative distance of a gold value but attached to unrelated keys.
  • Scale distractors are the same figure stated in a different unit.
  • Narrative distractors follow the semantic style of Breaking Focus.

Gold invariance is checked automatically: no inserted key tuple may equal a gold key tuple. A human audit of a sample checks that the question stays unambiguous. Each type is applied at several intensities, so we obtain a degradation curve for each type rather than a single score.

The second component is the address-and-execute reasoner. The model reads the linearized table and text. It outputs a short program over a FinQA-style operation set, whose arguments are cell addresses or text-span pointers plus a scale token. The model never outputs literal numbers. An executor resolves each address against the original document. The executor fails loudly when an address is ambiguous or does not exist, and the model is told about the failure.

Training proceeds in two stages, starting from Qwen2.5-7B and Qwen2.5-14B. The first stage is supervised fine-tuning on gold programs converted to addresses. The second is DPO. The preferred program binds the gold key, and the dispreferred program binds the colliding key produced by the generator for the same question. This makes the preference signal about exactly the discrete choice we expect to fail.

The third component is error attribution. Every wrong answer from every system is classified by replaying its program. The categories are wrong address (key-binding), right address but wrong digits or scale (value-transcription), wrong operation, or retrieval miss. This replay is what lets us test the first claim directly instead of inferring it from accuracy alone.

Baselines

Naive reference. The naive reference is the same base model, prompted with chain-of-thought over the full linearized context. We add the instruction to ignore irrelevant content, which Breaking Focus reports to be ineffective. This sets the floor that every intervention must clear on each distractor type.

Copy-based program generation. This baseline matches our model in every respect except one: its programs contain literal numbers copied from the context, in the style of FinQANet and AbstRaL's variable binding. It is trained with identical supervised data, identical DPO pairs and an identical tuning budget. This is the ablation that isolates addressing. Beating it means a lower value-transcription error rate together with equal or better clean accuracy. A gap under value-proximity and scale distractors is the specific signature we expect.

Generic distraction DPO. This applies the Breaking Focus recipe with untyped narrative distractors, using the same number of preference pairs and the same base checkpoint. It isolates the value of key-collision pairs. Beating it means a smaller accuracy drop under key-collision distractors at matched clean accuracy.

Relevance filtering. This baseline is CABINET-style soft relevance weighting, or Chain-of-Table pruning when CABINET cannot be re-run on our backbone. These methods are the strongest existing defenses against noise in tables. CABINET reports 69.1% accuracy on WikiTQ.

Strongest system. The strongest overall system is Orchestra, run with the same Qwen2.5-14B backbone. It reports 72.1% accuracy on WikiTQ with Qwen2.5-14B. We will re-run it on our perturbed splits and record token cost per query, because Orchestra relies on multiple agents and Monte Carlo voting.

Reference numbers for TAT-QA and FinQA. For TAT-QA, the only reported number in the file is TAGOP's 58.0% F1. For FinQA, the only one is FinQANet's 61.24% execution accuracy. No reported result was found on Lacuna for modern 7–14B LLMs on either benchmark, so we will produce those baselines ourselves under a shared protocol.

What counts as success. We count it as a win if both conditions hold. Clean accuracy must be within two points of the best baseline. Accuracy retained under the hardest key-collision setting must be substantially higher than every baseline, with the improvement attributable through the replay analysis.

Why not just…?

Why not just retrieve or filter better, as LDAR or CABINET do? LDAR learns how many passages to keep, and CABINET down-weights cells that look irrelevant. Both remove content that looks unrelated to the question. A key-collision distractor is, by construction, highly related: it shares the row label and differs only in year or segment. A filter that removes it risks removing the gold cell, which is the error-compounding problem CABINET itself documents for hard pruning. Filtering should help against narrative distractors. Our per-type curves will show whether it also helps against key collisions. If filtering closes the key-collision gap, that finding argues against our claim, and we will report it.

Why not just let the model write Python with the numbers in it? This is the real threat, and the copy-based ablation tests it directly. A strong model may simply copy digits correctly. If the replay analysis finds value-transcription errors rare under value-proximity and scale distractors, addressing contributes little and the proposal reduces to typed DPO. UDA's numerical collapse and TAT-QA's scale-prediction difficulty suggest the errors are not rare. That evidence is indirect, and the experiment settles the question.

Why not just use a long-reasoning model? Symbolic or Numerical? shows that DeepSeek-R1 avoids numerical noise by keeping quantities symbolic until the last step. It does so at an average of 14,698 tokens per answer. We include an R1-distilled model of matched size as a reference and report accuracy per token. The question is whether address-based reasoning gets the same protection at a small fraction of the cost.

Why not just run DPO on distracted examples, as Breaking Focus does? Generic distractors teach the model to resist narrative fluff. They may not teach it to tell "2019 revenue, segment A" apart from "2018 revenue, segment A". The generic-DPO baseline uses the same pair budget, so any advantage of typed pairs is measured rather than assumed.

Proposed Experiments and Decisive Checks

The diagnostic experiment comes first, because it can stop the project. We run the naive and copy-based systems on TAT-QA, FinQA and the FinHybrid and TatHybrid splits of UDA under each distractor type and intensity, and then attribute errors by replay. The first claim survives only if two things hold. Key-binding errors must dominate the degradation under key-collision distractors. Value-transcription errors must be non-trivial under value-proximity and scale distractors.

The main comparison trains the full system and all baselines on the TAT-QA and FinQA training splits. It reports F1 and execution accuracy on clean and perturbed test splits, together with the area under each per-type degradation curve and the number of tokens per query. We report WikiTQ and TabFact accuracy to check that addressing does not hurt general table reasoning. These two benchmarks have fewer arithmetic questions, so we expect smaller gains there.

Transfer is the decisive check against overfitting to our own generator. We train on generator-built key collisions from TAT-QA, then evaluate on naturally occurring confusable rows in UDA's full-document FinHybrid setting. There, the 45.9-versus-0.4 gap reported for GPT-4 (UDA) defines the regime we want to close. If gains appear on synthetic collisions but not on real long reports, we have learned the generator's artifacts and not robustness. In that case we will report the negative result rather than the in-distribution numbers alone.