Large language models are increasingly used as zero-shot regressors. A table row is written out as text, a few labeled rows go in the prompt, and the model is asked for the number. This is happening on public regression datasets such as the UCI collection and published algorithm-runtime logs, which have sat on the web for years. Every existing contamination detector was built for prose: it asks whether a sentence is suspiciously likely under the model. A memorized regression row looks nothing like that. It is a short string of digits whose "suspicious" part is a single target value. So nobody currently knows whether strong zero-shot regression scores on these benchmarks reflect reasoning from the features or recall of stored answers.
We propose two behavioral tests that need no access to training data.
The first is a target-recoding test. We re-express the target in the prompt on a secret scale, a random affine map of $\log y$, ask the model to predict on that scale, and invert. A model that reasons from the features loses nothing under this change, because the log scale keeps the multiplicative structure of skewed targets intact. A model that recalls memorized raw digits loses its advantage, because those digits no longer appear anywhere.
The second is an intervention-anchoring test borrowed from causal explanation. We change a feature in a suspected memorized row and measure how much the prediction moves. A model doing lookup stays pinned near the remembered answer and under-responds compared with how it treats fresh rows.
Together these give a detector for continuous targets. They also give a warning: interventional explanations computed on memorized rows describe the memory, not the model's learned relationship.
Core Claim
Memorization of a regression benchmark leaves two signatures that text-likelihood detectors cannot see and that can be measured purely from model outputs. The first is a performance gap between the raw target scale and a randomly recoded log scale. The second is damped responsiveness to feature interventions on member rows relative to non-member rows.
Both signatures should be measured in log-ratio error, the mean of $|\log \hat y - \log y|$. On heavy-tailed targets, raw squared error is dominated by a handful of huge values. That point is made in The When and How of Target Variable Transformations, which formalizes when to transform targets, and in the e-commerce sales study Analysis of E-Commerce Product Sales Prediction Based on Ga-Bp Neural Network. A raw-scale memorization advantage would therefore be noise-dominated.
The log transform plays two roles here. It is the recoding that a feature-based reasoner should be indifferent to, and it is the scale on which the contamination gap becomes measurable. Causal explanation supplies the second test. Zero-shot evaluation is the setting being audited.
Relation to Prior Work
Three lines of work motivate the gap.
Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? shows that near-perfect zero-shot causal scores on COPA collapse on the post-cutoff CausalProbe-2024. That is direct evidence that zero-shot scores on old public benchmarks are inflated. However, its benchmarks are all discrete multiple-choice, and its freshness check uses Min-K% Prob on text.
Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation audits perplexity against matched memorized and clean baselines, again for text tasks.
Can Tabular Foundation Models Predict Algorithm Runtime Distributions? shows that zero-shot in-context regression on log-scaled runtimes works remarkably well, but never asks about memorization.
The methods we reuse come from the contamination literature.
CAP: Data Contamination Detection via Consistency Amplification perturbs multiple-choice items in meaning-preserving ways (option shuffles, paraphrases) and flags models whose accuracy is brittle. Our target recoding is the continuous-target analogue of that idea, and CAP has no equivalent for numeric labels.
Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models normalizes each token's log-probability against the vocabulary distribution at that step. It is the strongest white-box text detector in the file. We apply it to serialized rows as a competitor.
DPDLLM: A Black-box Framework for Detecting Pre-training Data from Large Language Models classifies black-box completions with a small reference model. It is the strongest black-box text competitor.
The intervention test borrows from causal rationalization (On Causal Rationalization, Towards Trustworthy Explanation: On Causal Rationalization). That work defines explanations by whether flipping an input actually changes the prediction. We take the same move, intervene and watch the output, and repurpose it as a contamination signal for continuous outputs. We do not train selectors, so we avoid that method's token-flip cost and its monotonicity assumption.
One finding constrains the design. Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning shows that reasoning modes rationalize misleading context. We therefore run all probes both with and without step-by-step reasoning. We do not assume that chain-of-thought output reflects how the model actually computes.
Research Program
The model families are an open-weight LLM (a LLaMA-family model, matching the detectors' evaluation suites) and a closed API model, both used as serialized-row regressors.
TabPFN serves as a structural negative control. It is pretrained only on synthetic data, so it cannot have memorized UCI rows. Both of our signatures should read zero for it. To stay clear of the "context wall" that Can Tabular Foundation Models Predict Algorithm Runtime Distributions? reports, we keep contexts below 2,048 rows and avoid repeated instances.
The recoding test works as follows. For each dataset we draw a random scale $a$ and offset $b$ and present the targets as
$$z = a \log y + b.$$
The model predicts $\hat z$, which we invert to $\hat y = \exp((\hat z - b)/a)$. The recoding gap is the log-ratio error under recoding minus the log-ratio error on the raw scale. We average over several draws of $(a,b)$ so that no single code is learnable.
To separate "the model dislikes unfamiliar numbers" from "the model lost its memory," we use an ablation. We apply the same recoding to fresh data, rows the model cannot have seen, and subtract that gap. Only the excess gap counts as evidence of memorization.
The intervention-anchoring test works as follows. For a row $x$ and a feature $j$, we shift $x_j$ by a fixed quantile step and record the change in predicted log-target. The anchoring ratio is that change on candidate member rows divided by the same intervention's average effect on matched non-member rows with similar features. A ratio near 1 indicates feature-driven behavior. A ratio clearly below 1 indicates pull toward a stored answer.
This needs no ground-truth causal graph, because the reference is the model's own behavior on fresh rows. The cost is that the test measures anchoring, not causal correctness, and we say so.
Ground truth for memorization comes from two sources. The first is controlled contamination: we continue-pretrain the open model on serialized rows from a random half of each dataset's test split, at several repetition counts. That gives row-level member and non-member labels. The second is natural freshness for the runtime benchmarks: we re-run the same SAT solvers on the seven scenarios (Clasp-factoring through LPG-Zenotravel) with new seeds. The resulting runtimes did not exist before the model's training cutoff, which gives a clean split for the natural setting.
Baselines
The do-nothing reference is to report zero-shot log-ratio error and take it at face value. Every signature we propose is a correction to this number. A detector earns its place only if it changes conclusions relative to it.
The first real competitor is Min-K%++ applied to serialized rows, with $k$ tuned on the controlled-contamination validation split. It is the strongest reference-free white-box detector in the file. The file reports only relative gains: 6.2% to 10.5% AUROC over Min-K% on WikiMIA. That is text, and there is no reported result found on Lacuna for any detector on serialized tabular regression rows. We also run it with likelihood restricted to the target-value tokens, which is the most favorable version for it.
The second competitor is DPDLLM for the black-box setting. The file reports 71.4% AUC on WikiMIA2 for LLaMA-7B, versus 50–57% for perplexity and Min-K%. We train its reference classifier on the same member and non-member splits our method sees.
The third competitor is a CAP-style perturbation consistency test that uses only feature-side perturbations: shuffled column order and reworded column names, with no target recoding. This ablation isolates whether recoding the target, rather than perturbation in general, carries the signal.
All methods get the same rows, the same query budget per row (matched number of forward calls), and the same validation split for threshold tuning. Beating them means higher row-level AUROC on the controlled split at low repetition counts, which is the realistic regime. It also means a dataset-level verdict that matches the fresh-runtime ground truth where the competitors' verdicts do not.
Why not just…?
Why not just run Min-K%++ on the serialized rows? It may work at high repetition, and if it matches our AUROC across repetition counts we will report that recoding is unnecessary. The reason to expect it to fall short is that a row is mostly predictable formatting and digits. Min-K%++ itself notes it misses rare items that have not formed a sharp local peak in likelihood, and a dataset row seen a few times in a crawl is exactly such an item. The controlled-contamination sweep over repetition counts settles this directly.
Why not just evaluate on fresh, post-cutoff datasets? That is the right gold standard, and we use it for runtimes. But fresh regression data with the same features as legacy benchmarks is rare. Such data also goes stale once it is crawled, the obsolescence problem noted for CausalProbe-2024. More importantly, a fresh-data drop does not tell a practitioner which existing benchmark rows or explanations are compromised. Our tests audit the existing benchmark in place.
Why not just use tree ensembles or TabPFN, which cannot have memorized the data? For prediction alone, that is a fair recommendation. It is also why TabPFN is our negative control. The question here is whether published LLM-regressor results and explanations on public benchmarks can be trusted, and that cannot be answered by switching models.
Why not use Yeo-Johnson or quantile recoding instead of log? The When and How of Target Variable Transformations finds that these often beat the natural log for fitting. Our goal is different. We need a recoding that a reasoner should be indifferent to. A quantile code discards the magnitude spacing a model would legitimately reason with, so it would inflate the recoding gap even on fresh data. We include quantile recoding as an ablation, expecting a larger fresh-data gap.
Proposed Experiments
The data are the ten UCI regression datasets used in The When and How of Target Variable Transformations. These include Online News Popularity, with its heavy-tailed target, plus Auto MPG and Energy Efficiency as mildly skewed contrasts. The seven runtime scenarios, in their original and re-run versions, complete the set.
The first decisive check uses controlled contamination. We plot row-level AUROC for recoding gap, anchoring ratio, their combination, Min-K%++, DPDLLM, and feature-only consistency against repetition count (1, 2, 4, 16). The proposal succeeds if our combined signal is clearly above the text detectors at one to four repetitions. It fails if Min-K%++ restricted to target tokens matches it.
The second check is calibration on controls. TabPFN and the open model before contamination should show an excess recoding gap and an anchoring ratio indistinguishable from zero and one, respectively.
The third check is the natural audit. We compare excess recoding gaps on original versus re-run runtime data, and across UCI datasets ordered by web age.
The fourth check tests the consequence for explanations. For contaminated rows, we compare interventional feature attributions with those computed on matched fresh rows. We report how often the top-ranked feature changes.
Finally, we repeat every probe with reasoning on and off. If the reasoning mode hides anchoring by talking itself into the memorized value, that is itself a finding.
References
- The When and How of Target Variable Transformations
- Analysis of E-Commerce Product Sales Prediction Based on Ga-Bp Neural Network
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation
- Can Tabular Foundation Models Predict Algorithm Runtime Distributions?
- CAP: Data Contamination Detection via Consistency Amplification
- Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models
- DPDLLM: A Black-box Framework for Detecting Pre-training Data from Large Language Models
- On Causal Rationalization
- Towards Trustworthy Explanation: On Causal Rationalization
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning