Systems that mine thousands of papers into databases of reported results ask a large language model to read a paper and copy out numbers, dataset names, and model names. The model doing the reading was probably pre-trained on those same papers, and on the benchmark tables and leaderboards around them. So a record in the database may not come from the document at all. The model may be recalling what it saw during training, or what is typical for that benchmark. This is benchmark contamination turned on the extraction tool itself. It is invisible to the usual precision checks, because a remembered number is often the right number.
We propose to measure and flag this failure using the extractor's internal activations. For every extracted field, we read a "context-reliance" signal from the model's hidden state. This is a direction in activation space that separates answers grounded in the prompt from answers drawn from stored knowledge. We first build a counterfactual testbed that establishes ground truth: real arXiv sources whose tables are edited in controlled ways. We then ask whether a probe calibrated on a few hundred examples can catch memory-driven extractions in a single forward pass, and whether it does so better than the cheap checks a skeptic would reach for first. The payoff is practical. Living meta-analyses built on such databases, and the conclusions drawn from them, can be audited for whether they reflect the papers or the extractor's priors.
Core Claim
Instruction-tuned extractors will substitute remembered values for in-document values at a measurable rate. The rate will be higher for heavily cited benchmarks and models, and for papers published before the extractor's training cutoff. A linear context-reliance direction read from intermediate layers will flag these substitutions at the level of individual records. The case that matters is the one where surface checks fail: the remembered value also appears somewhere in the document, or the field is a name rather than a number.
Three lines of prior work motivate this claim. Context-Parametric Inversion shows that instruction fine-tuning can make a model ignore context that conflicts with its stored knowledge, even while benchmark scores rise. Its example is Llama2-7B, whose context reliance "increased from 30% to 60% initially with IFT, then dropped to around 35%." That paper measures the effect behaviorally on synthetic fact-conflict sets and never looks inside the model or at scientific extraction. Uncertainty as Feature Gaps supplies our method. It extracts context-reliance, comprehension, and honesty directions using representation engineering (contrastive prompts plus a principal-component step on hidden states), needs "as few as 64 to 256 labeled samples," and reports "up to a 13-point improvement in Prediction-Rejection Ratio over supervised baselines." It targets contextual question answering on Qasper, HotpotQA, and NarrativeQA, not extraction and not contamination.
On the literature-mining side, the LLMEVALDB pipeline in Can LLMs Help Uncover Insights about LLMs? supplies the application. It used Llama-3.1-70B-Instruct and GPT-4o to mine 18,127 records from 1,737 papers. It reports "100% precision for model names and metrics, and 95% for dataset names" on human validation, but has no test of whether values were read or recalled. AgentExpt and SurveyMaster share the same blind spot. SurveyMaster states that full-text claim verification at scale is currently unfeasible.
The closest existing work on contamination detection constrains the design rather than supplying it. CoDeC and CDD/TED from Generalization or Memorization detect contamination from output probabilities or from how similar repeated samples are. Both work at the dataset level and on benchmark items, not on extraction records. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that such output-level signals collapse to "50-58% AUROC" once models are trained on reasoning traces or with reinforcement learning. This is why we look inside the model, and why reasoning-distilled extractors are part of the test set.
Research Program
The first component is a counterfactual extraction testbed. It is the missing ground truth. We take arXiv LaTeX sources in the LLMEVALDB window (2023–2024) and apply three edits to results tables.
The first edit, value shift, replaces a reported score with a plausible nearby value that appears nowhere else in the paper. The second, cell swap, permutes values within a table so that every original number is still present in the document but attached to the wrong row. The third, name swap, exchanges dataset or baseline names between rows.
A faithful extractor must report the edited content. Reporting the original is a memory-driven substitution, and the edit tells us which fields are at risk. We pair each edited paper with its unedited version. We also stratify papers along two axes: likely-seen versus likely-unseen, meaning before or after the extractor's release, and famous versus obscure, meaning how often the benchmark and model are cited. The famous-versus-obscure axis uses the citation links in the AgentExpt knowledge base.
The second component is the probe. Following Uncertainty as Feature Gaps, we extract a context-reliance direction from contrastive pairs and score each extracted field by projecting the hidden state at its first token onto that direction. We compare two sources of contrastive pairs: generic knowledge-conflict data (CF_World_Facts, CF_Bio) and a small set of 64–256 scientific counterfactual records. The comparison tells us whether a general "reading versus remembering" direction transfers to table extraction.
On the Variability of Concept Activation Vectors warns that such directions shift with random seed and layer choice. We therefore report results across seeds and layers rather than a single best layer.
The third component is a debugging and audit layer. Flagged records are grouped into human-readable failure types, such as "famous benchmark, pre-cutoff, name field." This borrows the goal of error slice discovery in CB-SLICE, whose concept bottleneck we cannot use in a generative extractor. We then re-run the LLMEVALDB meta-analytic findings with flagged records removed or re-extracted. One such finding is the reported "+14.61 points mean improvement" from chain-of-thought prompting on math and symbolic reasoning. The question is whether these conclusions move.
Extractors are open-weight, since activations are required: Llama-3.1-70B-Instruct (the LLMEVALDB extractor), LLaMA-3.1-8B, Mistral-v0.3-7B, Qwen2.5-7B, and DeepSeek-R1-Distill-Llama-8B, which represents the reasoning-trained setting where the fragility result bites.
Baselines
The naive reference point is to trust the extractor, as LLMEVALDB does. We report its substitution rate on the counterfactual testbed. There is no reported result on Lacuna for memory-driven substitution rates in scientific extraction. The published figure closest to it is LLMEVALDB's 100% precision on model names and metrics, which was measured on unedited papers, where recall and reading cannot be told apart.
The strongest cheap competitor is source-span verification. It accepts an extracted value only if that exact string occurs in the LaTeX source, and flags it otherwise. It costs nothing and needs no model access. On value-shift edits it should be nearly perfect, because the original number is absent from the edited document. We run it on every record and report results separately by edit type. The probe beats it only if it wins on cell-swap and name-swap edits, where the substituted string is present in the document. We treat a gain of at least 10 AUROC points on that subset, with no loss on value-shift, as the bar.
The strongest method from the contamination literature is output-level confidence. We adapt it at the record level in two forms. The first is the token log-probability of the extracted field, plus the change in that log-probability when the paper is removed from the prompt, which follows the logic of CoDeC. The second is the output peakedness of CDD from Generalization or Memorization, computed over sampled extractions.
The reported numbers come from a different setting and are not directly comparable. CoDeC reports a dataset-level "AUC of 99.9% in separating seen from unseen data." CDD reports "relative improvements of 21.8% to 30.2% over existing baselines in Accuracy and AUC" on DETCON. No record-level extraction result was found on Lacuna for either method.
All detectors see the same records and the same extractor. Any method that uses labels gets the same 256 labeled counterfactual examples, and thresholds are tuned on a shared held-out split.
The ablations isolate the key ingredient in four ways. The first replaces the context-reliance direction with a random direction at the same layer. The second trains a supervised logistic probe on full hidden states with the same labels, which tests whether the semantic direction matters or any linear read-out works. The third uses the knowledge-conflict direction without scientific calibration. The fourth combines the probe with source-span verification, since the practical system would use both.
Why not just…?
Why not just check that every extracted string appears in the source? This is the real threat for numeric reversions, and we expect it to match the probe on value-shift edits. It cannot tell the difference between reading a number from the right cell and recalling a number that also appears in the paper, for example a prior-work score quoted in the text or a permuted cell. It says nothing about name fields that match a known benchmark list. The cell-swap and name-swap subsets exist to settle this question. If string checks match the probe there too, we will report that the activation signal adds nothing, and we will narrow the claim to non-verbatim fields.
Why not just perturb the paper at inference time and see whether the output follows the perturbation? This behavioral test is honest and model-agnostic, and it is a serious competitor. It requires a perturbation that preserves the truth, which is hard for tables, and at least one extra full pass over a long document for every record. We run it as an upper-bound baseline on a subset and report detection against compute. The probe earns its place only if it recovers most of that detection at the cost of a single forward pass.
Why not just discard papers from before the extractor's training cutoff? Cutoffs are often undisclosed or imprecise, and benchmark content is re-hosted after release (Test of Time). More basically, the purpose of mining is to cover the existing literature, most of which predates any current extractor. Our stratified testbed measures how much of the substitution effect a date filter would remove.
Why not have a second LLM verify each record? A judge model was likely trained on the same papers, so it shares the extractor's memory. ABCD: All Biases Come Disguised also documents systematic judge biases. We include a GPT-4o-style verifier on the counterfactual set to measure directly whether it endorses the remembered values.
Decisive Checks
The first question is whether memory-driven substitution exists at a rate that matters. It does if the substitution rate on famous, pre-cutoff papers significantly exceeds the rate on obscure or post-cutoff papers. If no extractor substitutes on more than about 1% of edited fields, the problem is too small to need a solution, and we report that as the main finding.
The second question is whether the probe beats string checks and output-confidence baselines on the cell-swap and name-swap subsets, and whether this holds for the reasoning-distilled extractor. That extractor is where the fragility result predicts output signals fail.
The third question is whether removing flagged records changes any headline LLMEVALDB meta-analytic estimate by more than its reported spread. That is the test of whether the audit matters for the conclusions people actually draw from these databases.
References
- Context-Parametric Inversion: Why Instruction Finetuning May Not Actually Improve Context Reliance
- Uncertainty as Feature Gaps: Epistemic Uncertainty Quantification of LLMs in Contextual Question-Answering
- Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMs
- AgentExpt: Automating AI Experiment Design with LLM-based Resource Retrieval Agent
- Automating Cognitive Distillation for Expert-Level Scientific Literature Synthesis
- Detecting Data Contamination in LLMs via In-Context Learning
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- On the Variability of Concept Activation Vectors
- CB-SLICE: Concept-Based Interpretable Error Slice Discovery
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- ABCD: All Biases Come Disguised