Small models trained specifically for tables, at 3B to 8B parameters, now report scores that match or beat GPT-4o on table question answering. These are tasks where a model reads a table, sometimes given as an image, and answers questions such as "what was the year-over-year change in operating income?" Every one of these results is measured on static public benchmarks such as WikiTableQuestions, TabFact, TAT-QA, HiTab and the MMTab suite. Those benchmarks have been on the web for years. The proposal is to build counterfactual twins of existing benchmark items. A twin keeps the real table's schema, layout and question. Its numbers are resampled under the table's own arithmetic constraints, and its answer is recomputed by executing a verified program. Because a twin looks and reads like the original but has a different correct answer, recalling the original answer stops helping. Asking it about the original layout also tells us whether the model can read the cells it needs.
The question we want to answer is simple. How much of the reported advantage of specialized table models survives when memorized answers are made useless, and when the model fails, is it failing to read the table or failing to reason over what it read? Text-only reasoning already has contamination-resistant generators, such as DyVal and BeyondBench. Both found that models scoring well on static sets drop to near zero on freshly generated problems. BeyondBench also found that math-specialized models did worse than their generalist base models. No comparable instrument exists for table reasoning, multimodal or otherwise. That is exactly where the strongest specialization claims are being made.
Core Claim
Our claim is that a twin gap separates three things for any table model: what it recalls from training, what it reads from the image, and what it can compute. The twin gap is accuracy on re-rendered originals minus accuracy on value-resampled twins, with the two drawn in identical visual style. Two further measurements complete the picture. A rendering gap is accuracy on the benchmark's own images minus accuracy on re-rendered originals. Cell-lookup probes attach to each twin and ask only "what is in row r, column c" for the cells the program uses.
We hypothesize that RL-specialized and SFT-specialized table models show a larger twin gap on the datasets they were trained near (held-in) than on held-out ones, and larger than their own base models show. If that holds, part of their reported advantage over frontier models is recall rather than skill. The opposite result would be equally informative: a specialized model whose gain survives intact on twins would be the first contamination-controlled evidence that table specialization teaches transferable competence. That would directly answer the tension BeyondBench raises.
The design choice that separates this from existing robustness tests is that a twin changes the answer. Row and column shuffling, as used by Beyond Linearization / TABGR and Reasoning-Table, leaves the gold answer unchanged. A model that memorized "the answer to this TAT-QA item is 12.4%" passes a shuffled version untouched. Swapping or removing chart titles, as in Plot Twist, removes one linguistic shortcut but not answer recall. Twins attack recall directly while holding the visual and linguistic surface fixed.
Research Program
Program recovery. For each item in TAT-QA, WikiTQ, HiTab, TabMWP and the MMSci-Eval split of Does Table Source Matter?, we recover an executable program that computes the gold answer from the table cells. TAT-QA and TabMWP often ship derivations. For the others, a strong language model proposes SQL or Python over a parsed version of the table, and we keep an item only if executing the program reproduces the gold answer. This execution-verification step is borrowed from CoReTab, which filtered its 115K training traces the same way. Here it is used to certify evaluation items rather than to make training data. Items with no recoverable program are dropped. We report the surviving fraction per dataset and question type, so that any selection bias toward easy arithmetic is visible.
Constraint-preserving resampling. This is the delicate part. Naive resampling produces tables that are implausible, such as totals that do not add up or percentages above 100. A model's accuracy could then drop because of distribution shift rather than lost memorization. We therefore infer constraints from each table: column data types, magnitude ranges and sign, row and column totals detected by checking sums, monotone time series, and percentage columns. We resample numeric cells within those constraints. Entity strings (company names, countries) are left alone in the main condition. In a secondary condition they are swapped for same-type entities, to see whether recall is keyed on entities.
We then re-execute the program to get the new answer. We reject twins whose answer coincides with the original or whose program now hits an edge case, such as a tie in an argmax question. Each item gets several twins. That lets us estimate per-item variance and generate fresh twins at evaluation time, so the instrument itself cannot be memorized.
Rendering and perception probes. Original and twin tables are both re-rendered as images through the same HTML renderer, sampling fonts, borders, gridlines and slight degradation. The gridline and plotting-style perturbations that hurt chart models in Plot Twist are thus sampled for both versions rather than confounded with the twin manipulation. Text-serialized versions (Markdown and HTML) are produced from the same parsed table, so text-only models such as Reasoning-Table and HInT can be evaluated on identical content.
For every twin, the cells the program references become cell-lookup probes. This follows Plot Twist's idea of testing elementary perception separately from the hard question, which the file notes has never been applied to tables. We then report twin accuracy conditional on passing all lookups for that item. That number is the model's reasoning accuracy given correct perception.
Baselines
The naive reference point is each model's reported score on the original static benchmark, re-measured with our own evaluation harness. The paper's headline comparison is how far a model falls from this number to its twin score, set against how far its base model falls.
The specialized competitors are the published table-specialized systems. Can GRPO Boost Complex Multimodal Table Understanding? (Table-R1) trains Qwen2-VL-7B with an SFT warm-up, then an RL stage that rewards correct reconstruction of table structure, then an RL stage for reasoning. It reports 48.06% accuracy on held-out TAT-QA against 37.82% for its SFT baseline. We also use the CoReTab-trained Qwen2.5-VL, which reports 97.14% on TABMWP and 81.57% on AIT-QA, and the MMSci-tuned Qwen2-VL-7B, which reports 42.10% table-QA accuracy and 73.98% fact-verification accuracy on MMSci-Eval. On the text side we use Reasoning-Table, a 7B model trained with GRPO using outcome, format and cell-citation rewards; it reports outperforming Claude-3.7-Sonnet by 4.0% and remaining stable under shuffling. We also use HInT, a 3B model that injects hypergraph features into the internal layers where the model represents rows and columns; it reports 67.39% exact match on WikiTQ against 65.44% for text-only SFT.
The strongest competitor on this specific question is Table-R1. It is the model whose RL recipe is most explicitly argued to produce reasoning rather than imitation, and it reports a large held-out gain. Its claim of real reasoning therefore deserves the hardest test. Where weights are not released, we retrain from the published recipe on the same base model, and we disclose that the result is a reproduction.
Every specialized model is paired with its exact base model (Qwen2-VL-7B, Qwen2.5-VL) and with GPT-4o. All are run with the same prompts, the same decoding (greedy, with no self-consistency sampling), the same image resolution, and no per-model prompt tuning beyond the format each model was trained to emit.
Specialization "survives" if the specialized model's twin accuracy beats its base model's twin accuracy by at least half of the margin it held on originals, with paired bootstrap intervals over items excluding zero. Specialization "is largely recall" if that margin shrinks by more than half while the lookup-conditioned reasoning accuracy of the two models converges.
The key ablation isolates the answer-changing ingredient. We compare twins that only shuffle rows and columns (answer preserved), twins that only resample values (answer changed), and twins that do both. If the twin gap appears only when answers change, the effect is recall, not sensitivity to layout.
Why not just…?
Why not just use the newer benchmarks, such as MMTABREAL or WildStruct? MMTBENCH reports 20–40% drops on its 500 real tables. TableMoE reports 80.31% on WildStruct. Both are newer than the training data of today's models, so they do help. But a drop on a new benchmark mixes three things: harder tables, distribution shift, and absence of memorized answers. Both are also static, so they inherit the same contamination problem once they are crawled. Twins hold difficulty and surface fixed and change only answer recall. We will also build twins of MMTABREAL items to test whether its reported drop is mostly difficulty, in which case its twin gap would be small.
Why not just run a contamination detector, such as n-gram overlap or membership inference? Overlap statistics need access to the training corpus, which is unavailable for GPT-4o and for the teacher-generated traces that Table-R1 and CoReTab trained on. Overlap also says nothing about whether the overlap actually raised the score. The twin gap measures the behavioural effect directly. We validate it with a planted-contamination experiment (below) that overlap methods cannot run on closed models.
Why not just generate fully synthetic tables, as DyVal does with random graphs? Synthetic tables remove contamination but also remove realism. A drop on them could simply mean the tables look unfamiliar. We include a fully synthetic arm built with the same renderer and question programs to show this confound. If models do worse on synthetic tables than on twins of real ones, the difference estimates the cost of unfamiliarity, which twins are designed to avoid.
Why not just shuffle rows and columns? Shuffling keeps the answer the same, so memorized answers still score. This is exactly the ablation above. Reasoning-Table's reported shuffling stability is therefore compatible with heavy recall, and our design tests that directly.
A real threat: do resampled values stay realistic enough? The main threat to the whole design is that resampling, even under constraints, introduces implausibility that lowers accuracy for reasons unrelated to memory. Our checks are a human plausibility rating on a sample of twins and the planted-contamination calibration. Most importantly, the twin gap for a model that cannot have seen the data should be near zero. If base models also show large twin gaps, the instrument is measuring distribution shift and we will report that rather than a contamination finding.
Proposed Experiments
Calibration. We first validate the instrument. We fine-tune Qwen2-VL-7B on the original items of a random half of a chosen TAT-QA evaluation split (deliberate contamination), leaving the other half unseen. A valid instrument shows a large twin gap on the seen half and a near-zero gap on the unseen half. We report the measured difference, together with the base model's gap on both halves.
Main study. We evaluate all models above under the three conditions (benchmark image, re-rendered original, re-rendered twin), plus text serializations for the text-only models, across TAT-QA, WikiTQ, HiTab, TabMWP and MMSci-Eval. We split by held-in versus held-out status according to each model's documented training data. The prediction from BeyondBench's result on math specialization is that the twin gap is largest for specialized models on their held-in sets.
Failure attribution. The perception probes split each failure into a lookup error or a reasoning error. This tests, for tables, the claim from MATH-Vision and GeoSQA that multimodal failures are mostly post-perception. If instead lookup errors dominate on twins of hierarchical HiTab tables, structural reading, the concern of the linearization line of work, is the bottleneck.
Constructive check. Because each twin comes with an executable verifier, the generator can also serve as an RL environment that supplies an unlimited stream of fresh, answer-verified twins. We train the base model with a GRPO-style setup on twins of training-split tables only, then measure whether this shrinks the twin gap on test-split twins relative to Table-R1's recipe at matched compute. Success would show that answer-changing training data is a cheap guard against the recall that static RL training may encourage.
The decisive outcome of the whole note is one number per specialized model: the fraction of its margin over its base model that survives on twins.
References
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
- Beyond Linearization: Attributed Table Graphs for Table Reasoning
- Reasoning-Table: Exploring Reinforcement Learning for Table Reasoning
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning
- CoReTab: Improving Multimodal Table Understanding with Code-driven Reasoning
- HInT: Hypergraph Infusion at the Structural Layers Improves Table Understanding
- Can GRPO Boost Complex Multimodal Table Understanding?
- MMTBENCH: A Unified Benchmark for Complex Multimodal Table Reasoning
- TableMoE: Neuro-Symbolic Routing for Structured Expert Reasoning in Multimodal Table Understanding
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level