Language models reason worse when a table is padded with rows and columns that have nothing to do with the question. This happens even when the answer is sitting right there. The standard explanation is that attention spreads a fixed budget of weight over every token, so each irrelevant cell takes a little weight away from the relevant ones. This note proposes a sharper version of that explanation and a test that can falsify it. Suppose the table reader attends within rows and within columns separately (dual-axis attention). Then an irrelevant column never competes with a cell's own column during the column step, and an irrelevant row never competes during the row step. That predicts a specific signature. For a dual-axis reader, the damage from extra rows and the damage from extra columns should add up independently. For a reader that sees the table as one flat sequence, the damage should depend on their product, the total number of distractor cells.
Testing this cleanly requires two things existing benchmarks lack. The first is tables where row noise and column noise can be dialed independently. The second is questions the model cannot have memorized. We therefore build a procedural generator. It writes fresh tables with invented entities, attaches a small program (filter, compare, aggregate) whose execution gives the ground-truth answer, and injects controlled distractors. We use the same program executions as a training signal: the model learns to predict what a program returns on a table, in the style of CodeI/O but applied to 2D grids. The generator therefore serves as both the contamination-free test bed and the source of training data.
Core Claim
A table reader that combines row-restricted and column-restricted attention layers with training on table-program input/output prediction will show separable robustness. In a regression of accuracy on distractor-row count $r$, distractor-column count $m$, and their product, the interaction coefficient will be near zero for this reader and significantly negative for an otherwise identical reader that sees a flattened table. A second prediction follows from the first. The advantage should be largest where distractor rows and columns co-occur. That combined regime is exactly what existing noisy-table benchmarks conflate and what external pruning handles by brute force.
The prediction comes from a simple counting argument, sketched here. In a flat sequence, each cell's softmax competes over roughly $(N+r)(M+m)$ tokens, where $N \times M$ is the clean table. In a column step it competes over $N+r$ tokens, and in a row step over $M+m$. We say "should" rather than "will" for two reasons. First, the top layers of our reader still use full attention so question tokens can reach every cell. Second, learned attention need not dilute uniformly. The regression test exists precisely to check whether the counting argument survives training.
Where This Sits
Dual-axis attention is well established for clean tabular data. SAINT alternates feature attention with attention across samples for classification. FlexTab and Orion-Bix use row-column attention for in-context tabular prediction. None of the three answers natural-language questions or studies distractors, and FlexTab itself flags possible contamination from its 300,000 crawled tables.
The closest mechanism is the Structure-Aware Transformer. It masks a single sequence so that lower layers attend within rows and upper layers across columns, reaching 73.23% on TabFact; removing the mask costs 5–9%. We reuse its masking approach because it needs no custom kernels. Our contribution is to measure how masking interacts with controlled noise, which that paper never does.
The noise literature motivates the question. UDA shows GPT-4 falling from 45.9 to 0.4 on FinHybrid when given full documents instead of retrieved chunks. Less is More shows narrative fluff hurting code synthesis. The tabular noise work constrains the design. EnoTab prunes up to 89.2% of large tables with an external evidence tree and gains 8.2–9.5% on STQA-N/L. It also needed an And2Or rollback because hard filtering emptied tables, which is a warning that deletion-based denoising drops needed premises.
The code-transfer line supplies the training objective. CodeI/O trains models to predict program outputs and inputs, raising Qwen 2.5 Coder 7B from 54.8 to 57.7 across 14 non-code benchmarks. It never touches grids or noise.
The program-based table reasoners are the main competitive threat. Table-r1 generates programs over flattened tables. CoReTab does the same while documenting a failure where models write code that simply prints a hallucinated answer.
The contamination work sets the evaluation standard. DyVal generates reasoning problems on the fly and finds that models strong on static tests score near zero. The 3-SAT study shows shortcuts collapsing at the hard phase transition. We carry this procedural logic into 2D tables, where no such generator is reported in the file.
Research Program
The generator samples a schema, fills rows with invented entities and typed values, and samples a program from a grammar of filters, comparisons, superlatives, counts, and single-hop lookups. It executes the program to get the answer and the set of touched cells, then renders a natural-language question from a template bank. Distractors come in three independent types. Row distractors are near-miss entities that nearly satisfy the filter. Column distractors have headers that are near-synonyms of the queried attribute, such as "revenue" versus "net revenue." Narrative distractors add backstory to the question. Test splits hold out program templates, entity vocabularies, and value ranges so that no test item overlaps training.
The reader is an open code model of roughly 1.5B parameters, with SAT-style masks applied to a linearized table. The lower two-thirds of the layers alternate between within-row and within-column masks, and the top third uses full attention. It is trained with two objectives. The first is table-program I/O prediction: given a table and a program, predict the output and the touched cells. The second is question answering: given a table and a question, predict the touched cells and then the answer. Because the model predicts outputs rather than emitting code, the trivial-printing failure cannot occur.
The central analysis fits accuracy $\approx a + b\,r + c\,m + d\,rm$ across a grid of $(r, m)$. Here $r$ is the number of distractor rows, $m$ the number of distractor columns, and $d$ the interaction coefficient the claim concerns. We also measure how much attention weight the touched cells receive, layer by layer, as distractors grow. This checks whether any robustness actually comes from the mechanism the counting argument names.
Baselines
The naive reference point is the same backbone, trained on the same generated data for the same number of steps, with no masks and a flat table. This is also the key ablation. If its interaction coefficient matches the masked reader's, the structural claim is false. A second ablation keeps the masks but replaces I/O prediction with answer-only supervision, isolating the contribution of the code-derived objective. Both ablations get the same hyperparameter search budget as the full model.
The strongest existing method on real tables is program-based reasoning in the style of Table-r1, which reports 76.5% on WTQ with Qwen2.5-Coder-7B. The same paper reports GPT-4o at 74.3% with programs and 63.2% with text reasoning. We will run the released Table-r1 approach, or a faithful reimplementation at 7B, on our generated splits and on WTQ and TabFact, together with GPT-4o in both program and text modes. On TabFact the reference is SAT's reported 73.23%. On noisy tables the reference is EnoTab applied to a frozen LLM; the file gives only its relative gain of 8.2–9.5% over Dater and Chain-of-Table on STQA-N/L, and no reported absolute number was found on Lacuna for it.
Beating these baselines means two things. The first is a flatter degradation surface, specifically a smaller interaction coefficient at matched clean accuracy. The second is higher absolute accuracy in the high-$r$, high-$m$ corner. A reader at 1.5B that merely matches a 7B program-based model on clean data but degrades less would count as a positive result. A reader that is flatter only because it is worse everywhere would not.
Why not just…?
Why not just have a strong LLM write a program over the table? This is the most serious threat. Once a program filters a dataframe, the interpreter makes distractor rows harmless. The weak point is the step before execution, where the model must map the question onto headers and cell values. Near-synonym columns and near-miss entities attack exactly that step. The evidence points both ways: UDA found text reasoning beat code interpreters on messy documents, while Table-r1 found programs beat text on WTQ. Our grid settles this per axis. We expect program-based models to be flat in $r$ and to degrade in $m$. If they are flat in both, we will report that the dual-axis advantage is limited to small-model efficiency and not accuracy.
Why not just prune the table externally first? EnoTab does this and works. But its need for rollback shows that pruning drops premises, and a pipeline decides relevance once, before any reasoning. We run EnoTab on the same grid. The comparison that matters is multi-condition questions, where the relevant cells are only identifiable jointly.
Why not just train the flat model on the same noisy synthetic data? This is a real threat. Data augmentation alone might teach robustness, and that is exactly why the unmasked ablation gets identical data and budget. If it closes the gap, the architectural claim fails. The I/O objective could still stand on its own, and the second ablation would show whether it does.
Why not just decontaminate TabFact and WTQ with n-gram filters? Semantic leakage survives surface filtering, as the near-zero-perplexity puzzles in Studying and improving reasoning in humans and machines and the DyVal collapses show. Static benchmarks also cannot vary $r$ and $m$ independently, and the claim depends on that. We still report static results, alongside "twins": real TabFact and WTQ schemas refilled with generated values and fresh programs. The accuracy gap between an original benchmark and its twin serves as a per-model contamination estimate.
Proposed Experiments
The decisive experiment is the $(r, m)$ grid on held-out generated splits, with clean tables of 10–40 rows, distractors up to three times the clean size on each axis, and five seeds. We compare interaction coefficients with bootstrap confidence intervals across the masked reader, the unmasked ablation, the answer-only ablation, Table-r1-style program models, GPT-4o in program and text modes, and EnoTab. Narrative noise is added as a third factor to check that the question-side dilution reported by Less is More behaves differently from grid-side dilution.
Transfer is tested on STQA-N and STQA-L, which contain naturally occurring noise, and on WTQ and TabFact, together with their twin versions. A mechanistic check tracks attention mass on touched cells as a function of $r$ and $m$.
The claim is supported if three things hold. The masked reader's interaction term is near zero while the unmasked reader's is clearly negative. The gap persists on STQA-N/L. And the masked reader's twin gap is smaller than that of the pretrained LLMs. The claim is refuted if the unmasked ablation trained on identical data shows the same separability.
References
- SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
- FlexTab: Towards a Flexible Encoder-Decoder Architecture for Tabular In-Context Learning
- Orion-Bix: Bi-Axial Attention for Tabular In-Context Learning
- Table Fact Verification with Structure-Aware Transformer
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- Less is More: Summary of Long Instructions is Better for Program Synthesis
- When TableQA Meets Noise: A Dual Denoising Framework for Complex Questions and Large-scale Tables
- CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction
- Table-r1: Self-supervised and Reinforcement Learning for Program-based Table Reasoning in Small Language Models
- CoReTab: Improving Multimodal Table Understanding with Code-driven Reasoning
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Can Large Language Models Reason? A Characterization via 3-SAT
- Studying and improving reasoning in humans and machines