Researchers who build systems that interpret tables, for example by expanding a cryptic header like esal into "employee salary" or by deciding that a column of numbers holds latencies and not accuracies, increasingly score those systems with a language model acting as a grader, called an "LLM judge." The judge has to see the table to decide whether "account balance" is an acceptable answer for acc_bal. Tables, however, are exactly the kind of input that is known to distract language models: long numeric grids, near-duplicate columns, and lookalike tables cause failures elsewhere. This proposal asks whether the judge inherits that weakness. Our hypothesis is that it does, and in a specific way. As irrelevant table context grows, judges become more lenient. They accept answers that are broader than required ("Organization" when the column holds technology companies) or answers that are plausible but wrong for this particular table. The result is that leaderboards for table interpretation systems quietly inflate.
We propose three things. The first is a human-verified meta-evaluation set for column-level interpretation, meaning a test of the graders rather than of the systems. The second is a controlled way to vary how much distracting table context the judge sees, so that judge accuracy can be plotted against the amount of noise. The third is a simple judging protocol that makes the judge commit to specific cell evidence before ruling. The protocol is a means of testing the diagnosis. The main contribution is the diagnosis itself and the benchmark that makes it measurable.
Core Claim
LLM judges for table semantics show leniency that depends on the amount of context. At a fixed amount of relevant signal, adding distractor columns, rows, or sibling tables raises the rate at which the judge accepts over-general or wrong-but-plausible candidates. Withholding the table altogether is not a fix. When the reference answer underdetermines correctness, as with valid alternative expansions or with units that are only recoverable from cell values, a judge that sees only the reference wrongly rejects good answers. The useful operating point lies between the two: the judge should see the evidence that bears on the column and nothing else.
How This Sits Relative to Prior Work
Several papers motivate the claim by showing that table context distracts models when they generate answers. UDA benchmarks retrieval and long-context models on real documents with multi-page tables. Its long-context GPT-4 scores 0.4 on FinHybrid numerical reasoning, against 45.9 with retrieval. CoDA-Bench places code agents among roughly 980 distractor files and finds they often run correct code on the wrong lookalike table. Accurate Table Question Answering with Accessible LLMs splits table QA across separate agents to shield small models from mixed prompts, and lifts Mistral-7B on WikiTQ from 3.6% to 45.9%. All three study the model that produces answers. None asks what happens when the grader faces the same input.
Two papers supply the meta-evaluation methodology, and one of them constrains our design. Evaluating Open-QA Evaluation builds EVOUNA, a human-labeled set for grading the graders. It reports that GPT-4 reaches 96–98% agreement on a TriviaQA subset and that weaker judges overgeneralize. Its inputs are isolated text pairs with no structured context, so the high agreement it reports is the number our hypothesis predicts will erode on tables. VALOR-EVAL parses outputs into structured tuples before a verification step. Aligning Extraction and Generation (Ext2Gen) inserts an explicit extraction step that removes noise before generation. Our evidence-first protocol borrows this "extract, then decide" structure and applies it to the grader instead of the generator.
The tabular-ambiguity papers supply the tasks and the candidate systems. Columbo expands abbreviated column names. It released real enterprise and science datasets (Finance, University, Adventure Works, EDI) and replaced exact match with a synonym-aware programmatic metric. That metric is the closest existing evaluator for this setting, and it never looks at cell values. Frozen LLM Column Embeddings (ColEmb) classifies column types from frozen Mistral-7B states over 25 sampled cells, reaching 90.2 average micro-F1. Results of SemTab 2023 scores column type annotation by strict knowledge-graph identifier match and reports that the food-composition set tFood is especially hard. Strict identifier matching is the rigid evaluator that motivates semantic judging. The overgeneralization failure in EVOUNA is the reason that move is risky.
On data construction, we reuse the author–reviewer pattern from Bae et al., where a model drafts and humans filter, which that work found about 13 times cheaper than writing from scratch. We also heed EHRSQL's warning that automated paraphrasing smooths away domain shorthand, so humans, not models, decide correctness.
Research Program
The benchmark covers two column-interpretation tasks that existing datasets already ground. The first is abbreviation expansion on the four Columbo real-world datasets plus NameGuess. The second is column type annotation on SOTAB, T2D, and SemTab tFood. For each column we collect candidate answers from real systems: a Columbo-style multi-stage expander, a direct-prompted open-weight LLM, ColEmb, and a fine-tuned baseline where one is available.
Real system outputs may be too easy to grade, so we add model-in-the-loop near-misses. An LLM is instructed to produce four kinds of candidates for each column: a hypernym of the true type, a sibling type, an expansion that is plausible for the header string but contradicted by the cells, and a valid but non-reference synonym. Annotators then label every candidate as correct or incorrect. Annotators always see the full table, and a second annotator labels a subset so we can compute agreement. EVOUNA's human kappa of 86.4–100 is the bar we aim to match.
Annotators also tag each item as either "reference-sufficient" or "context-required." An item is context-required when the reference alone cannot settle correctness, for example when a valid alternative expansion or a unit decision depends on cell values. This tag is what lets the study separate harm caused by noise from benefit provided by context.
Context is varied along a dose axis while the relevant signal stays fixed. The judge always sees the target column header, its cells, and the reference. To that we add, in increasing amounts, other columns from the same table, extra rows that include numeric outliers, and finally sibling tables drawn from the same data lake or Kaggle community. CoDA-Bench's lookalike-file trap is the template for this last step. Judges are open-weight models (Mistral-7B, Qwen2.5-14B, Llama3.1-70B) so that results are reproducible despite API drift, plus one frontier API model as a reference point.
The evidence-first protocol runs in two calls. In the first, the judge sees the full context and must quote the specific header tokens and cells that bear on the column's meaning. In the second, the judge sees only those quotes, the reference, and the candidate. It must classify the candidate as equivalent, narrower, broader, or different, and only "equivalent" counts as accepted. The explicit "broader" option is meant to address overgeneralization.
Baselines
The naive reference point is exact match against the reference, and for column types, SemTab-style identifier match. We run it on every item at no cost. It defines a floor for false rejections, and its false-accept rate is near zero by construction. Any judge that fails to reduce false rejections on the synonym items has added nothing.
The strongest existing programmatic evaluator is Columbo's synonym-aware accuracy. For expansion, we reproduce it on the same items. For types, we adapt it with ontology synonyms. The comparison point from the file is Columbo's own exact-match accuracy of 72.6–93.3% on its new datasets (versus NameGuess at 63.2–73.8%), which is a system score. We found no reported result on Lacuna for any evaluator's agreement with human correctness labels on column interpretation. That agreement is what we will measure, and the evidence-first judge must beat the synonym-aware metric on human agreement, especially on the context-required items.
The strongest LLM-judge baseline is a single-call judge given the full table context. This is how practitioners would naturally use a judge, and it is the setting EVOUNA validated on text, where GPT-4 reached 96–98% (EVOUNA, TriviaQA subset). Its counterpart is the reference-only judge, which sees no table at all. Both use the same models, temperature zero, and the same prompt-tuning budget: five prompt variants each, chosen on a held-out tenth of items.
Two ablations isolate the key ingredient. The first gives the second call randomly chosen cells of equal length instead of the quoted evidence, which tests whether the selection of evidence matters or only its brevity. The second removes the four-way equivalent/narrower/broader/different rubric, which tests whether the anti-overgeneralization framing matters separately from removing context. Beating the full-context judge means a lower false-accept rate on hypernym and contradicted-expansion items at the highest noise level, with no rise in false rejections. Beating the reference-only judge means fewer false rejections on context-required items.
Why not just…?
Why not just use a stronger judge, since GPT-4 already matches humans? EVOUNA's agreement figure comes from short text pairs with no structured context. UDA shows that the same model family collapses numerically when given whole tables, so strength measured on clean inputs does not carry over. This is nonetheless a real threat. If the frontier judge's curve is flat across noise levels, the leniency hypothesis fails for strong judges, and we would report it as a problem limited to small judges. The dose curves across four model sizes settle this question directly.
Why not just withhold the table and judge against the reference only? This is the most serious alternative, because it removes noise for free. Its weakness is underdetermination. Columbo needed a synonym-aware metric precisely because valid expansions differ from references, and some of those differences can only be adjudicated by looking at cell values. The context-required split measures this directly. If reference-only judging matches the evidence-first protocol on that split, we will conclude that context should simply be withheld, and that conclusion is publishable and useful in its own right.
Why not just extend Columbo's programmatic synonym matching, or use embedding similarity? Programmatic matching is reproducible and cheap. It cannot, however, tell that "account balance" is wrong for acc_bal when the cells are timestamps. EVOUNA found BERT-Score the worst evaluator because extra content dilutes its embeddings. We include the synonym-aware metric as a baseline, and if it ties the judges on reference-sufficient items, we recommend it for that portion of the data.
Why not just generate the near-miss candidates fully automatically? Generation is automated in our design. Labeling is not, because the evaluated quantity is agreement with human correctness, and an LLM-labeled gold standard would be circular. BrokenMath's sycophancy results (Petrov et al.) and EHRSQL's jargon smoothing both argue against letting models certify their own artifacts.
Decisive Experiments
The first test is the dose curve. For each judge, we plot false-accept rate on hypernym, sibling, and contradicted-expansion items against noise level, with the relevant signal held fixed. A positive slope that is significant across items, tested with a bootstrap over columns, confirms the core claim. A flat curve refutes it. We expect the slope to be steepest on numeric columns and on tFood, given UDA's numerical collapse and SemTab's tFood difficulty. These expectations come from the cited results rather than from any theory.
The second test is the crossover. On context-required items, the reference-only judge should show high false rejection, while the full-context judge should show rising false acceptance as noise grows. The evidence-first judge should dominate both only if neither ablation, random cells or no rubric, reaches its performance.
The third test is downstream consequence. We rank the candidate systems (Columbo-style expander, prompted LLM, ColEmb) under human labels and under each judge, and report rank correlation. This measures whether context-induced leniency changes which system a field would declare best. If the rankings agree despite the judge's errors, the practical stakes are smaller than claimed, and we will say so.
References
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- CoDA-Bench: Can Code Agents Handle Data-Intensive Tasks?
- Accurate Table Question Answering with Accessible LLMs
- Evaluating Open-QA Evaluation
- VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Columbo: Expanding Abbreviated Column Names for Tabular Data Using Large Language Models
- Frozen LLM Column Embeddings Are Strong Baselines for Column Type Annotation
- Results of SemTab 2023
- Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models
- EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records
- BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMs