Testing for Skill Beyond Recall: Conditional Independence Audits of Contaminated Benchmarks

When a language model scores well on a public benchmark, we usually cannot tell whether it solved the problems or remembered the answers. Recent work showed a simple way to ask this. First, measure what the model can recall about each benchmark item directly. Then check whether its answers still track the truth once that recall is subtracted out. On public economic time series, this subtraction erased almost all apparent forecasting skill. The subtraction, though, was a straight-line regression, and the "skill that survives" was a raw correlation with no error control. We propose to recast the question as a formal statistical test. Genuine skill means the model's answer carries information about the correct answer beyond what its recall explains. Pure memorization means it does not, and that is a conditional independence statement that tools already exist to test.

Concretely, we treat "the model's answer $S$ is independent of the true answer $Y$ once we know its recall and the item's intrinsic difficulty $Z$" as the null hypothesis. We test it with the generalised covariance measure. That measure checks whether the leftover errors of two flexible predictions, one predicting $S$ from $Z$ and one predicting $Y$ from $Z$, are correlated. We build recall probes that work for multiple-choice and math word-problem benchmarks, not just dated numeric series. We calibrate everything on models where contamination is planted by fine-tuning, so the true amount of skill is known. The output is a p-value and a decontaminated skill estimate for each model–benchmark pair, and it needs only text access to the model.

Core Claim

The closest existing work, Recall Residualisation, prompts a model for the exact historical value of a series ($R$). It elicits an indirect forecast ($S$), regresses $S$ on $R$ by ordinary least squares, and reports the correlation of the residual with the truth. Their headline results are that correlations as high as 0.93 fell to 0.17 or lower, and that LeakShare, the share of apparent skill explained by recall, was at least 0.97 in 19 of 20 cases. The authors acknowledge that a linear fit may miss nonlinear contamination.

Our first claim is sharper than "use a nonlinear regressor." Under the pure-recall null, any part of the recall-to-answer relationship that the regression fails to capture stays in the residual. Because recall is itself correlated with the truth, that leftover recall shows up as spurious "skill." Underfitting therefore biases linear residualization toward overstating genuine ability, and a correctly calibrated test should find less surviving skill wherever recall acts nonlinearly. The single surviving GISTEMP correlation of +0.17 is a natural first case to recheck.

Our second claim is that the approach transfers to categorical and short-answer benchmarks such as MMLU, ARC, HellaSwag and GSM8K. Two things make the transfer work. We define recall probes that do not require the full item, and we add item-difficulty covariates to the conditioning set. The difficulty covariates matter because items a model fails to recall are not a random subset.

Research Program

The test statistic follows Shah & Peters. We fit $\hat f(Z) \approx E[S \mid Z]$ and $\hat g(Z) \approx E[Y \mid Z]$ on one data split. On the other split we form per-item products $r_i = (S_i - \hat f(Z_i))^\top (Y_i - \hat g(Z_i))$ and compute

$$T = \frac{\sqrt{n}\,\bar r}{\hat\sigma_r},$$

where $\bar r$ and $\hat\sigma_r$ are the sample mean and standard deviation of the $r_i$ over $n$ held-out items. The splits are then swapped and the results combined. $T$ is approximately standard normal under the null, provided the product of the two regression errors shrinks faster than $1/\sqrt{n}$; that condition is the source of the test's validity guarantee.

For multiple-choice items, $S$ and $Y$ are one-hot vectors over the options, and the regressions are classifiers. This follows the residual construction for categorical data in Ankan & Textor, which avoids the empty-cell problem of contingency tables. For numeric answers, as in GSM8K and the time series, we rank-transform before regressing. The decontaminated skill estimate is the normalized residual covariance, and our generalization of LeakShare is one minus the ratio of residual to raw covariance.

The main engineering work is the recall vector $R$. For language benchmarks we use four probes, each of which withholds information needed to reason out the answer. The first asks for the answer given only a truncated prefix of the question. The second asks for the answer given only the option list. The third prompts with benchmark metadata, of the form "MMLU college chemistry, item 37, answer:". The fourth scores the overlap of a verbatim continuation of the question stem, the instance-generation signal catalogued by Fu et al.. Where log-probabilities are available, each probe yields a graded confidence instead of a single guess.

Difficulty covariates $D$ come from the item itself: subject, length, number of reasoning steps for GSM8K, and number of distractors. We deliberately avoid scores from reference models, because the file documents that whole model families may be contaminated on the same benchmark. Together, $Z = (R, D)$ has roughly ten to thirty dimensions of mixed type. Stratifying on $Z$ breaks down at that size, which is exactly the regime where residual-based tests are meant to operate.

Two known limits shape the design, and we state them openly. The first is the hardness result of Shah & Peters, reinforced in practice by On the Hardness of Conditional Independence Testing In Practice: no conditional independence test is valid without assumptions. Ours are regression-rate assumptions on a low-dimensional $Z$, and we check them empirically on known-null controls rather than simply assert them.

The second limit is that the test can only detect memorization that passes through the probes. If a model memorized paraphrased items, as in the evasion attack of Dekoninck et al., and none of our probes detects it, the test will attribute that memorization to skill. We measure this failure directly rather than assume it away. In the other direction, a partial-question probe may sometimes succeed by reasoning, which would absorb genuine skill into $R$. That bias makes the test conservative about claiming skill, which is the right direction for an audit.

Baselines

The naive reference point is raw benchmark accuracy, or the raw correlation between $S$ and $Y$, with no adjustment. We report it for every model–benchmark pair because every other number is measured as a change from it.

The most direct competitor is linear recall residualization exactly as in Recall Residualisation. This means OLS of $S$ on the same $R$, followed by the correlation of the residual with $Y$. On categorical benchmarks we extend it naturally with a linear probability model. It is run on identical probes, items and splits, so that the only difference is the regression and the inference procedure. On the four time series we first reproduce its reported figures: correlations of up to 0.93 falling to at most 0.17, LeakShare of at least 0.97 in 19 of 20 cases, and +0.17 surviving on GISTEMP.

Beating this baseline has a precise meaning. On planted-contamination models with known ground truth, our skill estimate should have lower error. On negative controls, our Type I error should stay at the nominal 5% where the linear method's inflates.

The strongest existing detector that needs no training data is ConStat (ConStat). It compares performance on original versus rephrased items against a panel of reference models. It reports a contamination gap of $\delta \approx 8.3\%$ for Mistral-7b-v0.1 on GSM8K and $\delta \approx 6.6\%$ for Llama-3-70b on ARC. We rerun ConStat on the same target models with its reference panel, and we match inference budgets by counting total model queries.

We compare two quantities. The first is our model-level estimate of accuracy lost to recall. The second is ConStat's $\delta$, which measures the same contamination-driven inflation from a different angle. On planted models, the method whose estimate lands closer to the true inflation wins.

PaCoST (PaCoST) is the relevant dataset-level significance test. We compare its detection rate against ours at matched false-positive rates on open models where log-probabilities are available. There is no reported result found on Lacuna for PaCoST's numerical detection power; it reports only significance at $p<0.05$. Perplexity-based membership inference serves as a floor. Fu et al. report an AUC of 99.4 on fine-tuning contamination but roughly 0.50 on pretraining corpora.

Two ablations isolate our key ingredients. The first replaces the flexible regressions with linear ones while keeping the full test machinery. This separates the effect of the nonlinear fit from the effect of formal inference. The second drops the difficulty covariates $D$ from $Z$, which checks whether they are needed to avoid selection bias.

Why not just…?

Why not just stratify accuracy by whether the recall probe succeeded? If $R$ were a single binary "recalled the answer" flag, stratifying would be an exact test, and our method would add little. That scenario is a real threat, so we include it as a baseline. It breaks down in two ways once $Z$ grows. With four graded probes plus difficulty features, most strata contain few items. And items that fail recall differ systematically in difficulty, so the within-stratum accuracy of unrecalled items mixes skill with item selection. The difficulty-ablation experiment measures how large that distortion is. If stratification on binary probes matches our planted-contamination error, we will report that the simple method suffices for that benchmark.

Why not just use ConStat's reference-model comparison? ConStat estimates dataset-level inflation and depends on a clean reference panel. PaCoST's finding of contamination across the Llama, Mistral and Qwen families suggests such a panel may not exist for older benchmarks. Our test uses only the target model's own probes, gives item-level residuals, and returns a calibrated p-value. The head-to-head on planted models, where the true inflation is known, settles which estimate is more accurate.

Why not just retire static benchmarks for fresh or procedurally generated ones like A²RBench? Fresh benchmarks prevent future contamination but say nothing about the historical scores on MMLU, ARC or GSM8K that the field still compares against. They are also hard to build for expert domains. We view them as complementary. In fact, we use freshly generated arithmetic items as a positive control where skill is known to be genuine.

Why not just add white-box activation features, such as the memorization subspaces of A Narrowing Geometry in Contaminated Reasoning, as the recall signal? Those features may enrich $Z$ for open models, and we test this as an optional extension. They are unavailable for the frontier API models that most need auditing, so the core method stays black-box.

Proposed Experiments

The decisive calibration experiment uses planted contamination. We fine-tune open models, including Pythia and OLMo-2 checkpoints and Mistral-7B, on known random subsets of MMLU, ARC, HellaSwag and GSM8K. Contamination is planted three ways: verbatim, paraphrased in the style of the evasion attack of Dekoninck et al., and mixed. The un-fine-tuned model's accuracy on each item defines the true skill, and the gain after contamination defines the true inflation.

We measure four things. The first is the error of each method's skill estimate. The second is power as the contaminated fraction varies. The third is Type I error on a fabricated negative-control set: MMLU-style questions whose answers are assigned at random and memorized during fine-tuning. On that set skill is impossible by construction, so the null holds exactly, mirroring the fictional-factor controls of Recall Residualisation. The fourth is a positive control on freshly generated multi-step arithmetic in the style of MSCalc from A Narrowing Geometry in Contaminated Reasoning, where the test should find skill.

Several outcomes would be decisive. The proposal succeeds if the test keeps nominal size on the fabricated set while linear residualization over-reports skill there. It should also recover the true inflation more accurately than ConStat on verbatim contamination. The key failure we will report as a finding, not a footnote, is how much paraphrased contamination escapes the probes; the attack reports up to 15% score inflation with detection near 1%.

We then replicate the four time-series audits on Fama–French Mkt-RF, FRED UNRATE, FRED CPI YoY and NASA GISTEMP, adding a nonlinear fit and formal p-values. This checks whether the surviving GISTEMP signal is real skill or leftover recall. Finally, we audit public open and API models on the four language benchmarks. For each model–benchmark pair we report raw accuracy, decontaminated skill with confidence intervals, and our generalized LeakShare, and we compare the flagged pairs against ConStat's reported gaps.