Memorized Confidence: How Benchmark Contamination Inflates Cascade Savings

A common way to cut the cost of large language models is a cascade. A small model answers first, and the query goes to a larger, more expensive model only when the small model is unsure. Published cascades report large savings, for example 89% on QQP and up to 98.3% on HEADLINES. Those savings are measured on public benchmarks that the models have probably seen during training. When a model has memorized a question, its confidence is high because of recall, not because it can reason about the question. A cascade tuned on such data learns to trust the small model too much. The same flaw affects label-free monitors that estimate deployed accuracy from average confidence.

This note proposes to measure how much of the reported efficiency comes from memorization. It also proposes a cheap fix. The fix is a per-query "was this probably memorized?" signal: how much the small model's confidence changes when the answer options are reshuffled. That signal is fed into a learned deferral rule and into the accuracy monitor, so both keep working when deployment traffic contains fresh questions.

Core Claim

We make three claims, each testable.

First, on contaminated benchmarks the relationship between a small model's confidence and its correctness differs from that relationship on fresh questions of the same topic and format. As a result, a deferral threshold or gatekeeper tuned on one transfers poorly to the other.

Second, the size of this effect depends on the traffic mix. If deployment traffic mixes memorized and fresh queries in unknown proportions, no single threshold is right for every mix, even one tuned on clean data.

Third, adding an instance-level memorization feature to a learned deferral rule recovers most of the lost accuracy-per-cost, at the price of one extra small-model forward pass. The same feature repairs label-free accuracy estimates.

What is new is treating contamination as a failure mode of cost-aware inference. Prior work treats it only as a problem for offline evaluation integrity.

Where This Sits

Three lines of cascade work supply the machinery, and all of them tune and evaluate on static public data.

  • Model Cascading escalates BERT variants on fixed confidence thresholds across GLUE-style tasks.
  • FrugalGPT routes across commercial APIs using a DistilBERT answer scorer.
  • When Does Confidence-Based Cascade Deferral Suffice? shows that the Bayes-optimal rule depends on the difference in correctness probabilities between the two models, not on the small model's absolute confidence. It then trains a small MLP to predict that difference on ImageNet and CIFAR-100, and names specialist, noise, and distribution-shift failure modes.

We reuse that paper's method. Our change is to add a fourth failure mode, memorization, and to give the gatekeeper a feature aimed at it.

Two further papers constrain the design:

Both argue against "just calibrate" as the answer. Both are vision-only.

The contamination literature motivates the problem.

  • PaCoST detects contamination by testing whether confidence is significantly higher on original than on rephrased questions. It found significant contamination in Llama, Mistral, Phi-3 and Qwen models on MMLU, ARC, HellaSwag and others. It works only at the dataset level and needs a helper model to rephrase.
  • MMLU-CF reports an average drop of roughly 14% across 40 models, with o1 falling from 92.3% to 80.3%.
  • Meta Probing Agents reports a 15.7% drop for GPT-4-Turbo on probed MMLU.

We borrow PaCoST's intuition, that memorization shows up as sensitivity to surface form. We make it per-instance and cheap enough to run inside an inference path.

The monitoring literature is what breaks. Confidence-based Estimators for Predictive Performance in Model Monitoring proves that average confidence is an unbiased accuracy estimate for a calibrated model under covariate shift. Contamination violates the premise: memorized items are calibrated differently from fresh ones. Plug-and-Play Performance Estimation maps token log-likelihood features to task accuracy. Log-likelihood features are exactly what memorization distorts, as the near-zero surprise on classic puzzles in Studying and improving reasoning in humans and machines illustrates.

Research Program

Model ladder

We work with open-weight model families that expose token probabilities and that PaCoST found contaminated. A ladder of sizes is taken within one family, so the backup model's pre-training exposure overlaps with the small model's.

Signals and the gatekeeper

For each query $x$ we compute two kinds of signal.

  • Confidence features. The small model's confidence features $s(x)$, following Jitkrittum et al.: entropy and top-k answer probabilities.
  • Memorization score $m(x)$. For multiple-choice data, this is the average absolute change in the small model's probability on its chosen answer when the options are randomly permuted, plus whether its argmax answer survives the permutation. This needs one batched extra pass and no helper model.

The gatekeeper is a small MLP that estimates the headroom:

$$h(x) = P(M_2 \text{ correct} \mid x) - P(M_1 \text{ correct} \mid x).$$

Here $M_1$ is the small model and $M_2$ the large one. The gatekeeper computes this from $s(x)$ and $m(x)$, and the query is deferred when the estimate exceeds a cost parameter $C$ that we sweep. The gatekeeper is trained only on clean data (the MMLU-CF validation split) and tested on held-out clean and contaminated data.

Label-free accuracy estimate

For accuracy monitoring, we replace plain average confidence with an average of confidences recalibrated conditional on the memorization score. We keep the Poisson-binomial confidence intervals from Kivimäki et al.

Ground truth for contamination

Natural contamination is uncertain at the instance level, so we also build controlled contamination following PaCoST's protocol. WMDP was released as a clean, post-cutoff benchmark. We fine-tune the small model, the large model, or both on a random half of WMDP questions. This gives exact per-item labels and lets us create the asymmetric case the cascade literature has not considered, where the backup model has memorized items the small model has not.

Baselines

Naive reference points. The first is always calling the large model, which sets the accuracy target and the cost ceiling. The second is a max-probability threshold tuned on the original MMLU validation split, which is how a practitioner would tune on the benchmark at hand. This naive cascade is exactly the object whose savings we claim are inflated.

Distance-to-uniform threshold. This is the better confidence score from Model Cascading. That paper reports matching the standalone large model's accuracy at 11.07% of the computation cost on QQP with three models. We tune its threshold on the same splits as every other method.

Strongest existing method. This is the post-hoc headroom gatekeeper of When Does Confidence-Based Cascade Deferral Suffice?: the same MLP and confidence features, no memorization feature. It is the strongest competitor because it already estimates headroom, which is the Bayes-optimal target, rather than raw confidence. The file gives no scalar accuracy or compute delta for its rule, so we have no reported number to quote, and on LLM cascades over MMLU or MMLU-CF there is no reported result on Lacuna.

We run it with the same training data (MMLU-CF validation), the same architecture, the same hyperparameter search budget, and the same cost sweep. Its only difference from our method is the absence of $m(x)$. Beating it means a higher area under the accuracy-versus-cost curve on the clean test sets, with non-overlapping bootstrap intervals. It also means lower variance of that area across contamination mixes.

Isolating the key ingredient. Two ablations do this. The first swaps permutation sensitivity for Min-k% Prob on the question tokens, which needs no extra pass. The second replaces $m(x)$ with a random feature of matched dimensionality, to rule out gains from capacity alone.

Monitoring baselines. These are plain average confidence and ATC, both following Kivimäki et al., and the XGBoost log-likelihood meta-model of Plug-and-Play Performance Estimation. That meta-model reports 31.1% lower estimation error than ATC on CrossFit with OPT-6.7B.

Why not just…?

Why not just tune the threshold on a clean dev set like MMLU-CF validation? This is the most serious threat, and it may partly succeed. If deployment traffic were uniformly fresh, a clean-tuned threshold might be enough, and the contribution would shrink to a measurement plus a best practice. Our bet is that real traffic mixes memorized and fresh queries. A single threshold is then set for one mix and wrong for others, because the two populations have different confidence-to-correctness curves.

The mixture-sweep experiment below tests this directly. We include the clean-tuned threshold as a baseline. If its accuracy-per-cost area stays within noise of our gatekeeper across all mixes, we report that honestly and drop the gatekeeper claim.

Why not just calibrate the small model with temperature scaling or isotonic regression? A single temperature rescales all confidences monotonically, so it cannot separate a confident memorized answer from a confident wrong one. Learning to Cascade and Rethinking Calibration for Early-Exit Neural Networks both show that calibration is the wrong target for staged inference. We still run temperature and isotonic calibration on MMLU-CF validation, since their evidence is from vision.

Why not just filter contaminated items offline with n-gram overlap or Min-k% Prob? Offline filtering cleans a benchmark, not a live query stream. PaCoST reports that Min-k% Prob is less sensitive than paired-confidence testing, and MMLU-CF shows memorization that survives rephrasing. Still, Min-k% Prob computed at inference time costs nothing extra, so it appears as the ablation feature above. If it matches permutation sensitivity, we adopt it, because it is cheaper.

Why not just always call the large model? The large model is contaminated too; o1 loses 12 points on MMLU-CF. Cascades also sometimes beat the large model, by 2.18% on CommitmentBank in Model Cascading. Always-large serves as the cost ceiling and accuracy target in every plot.

Proposed Experiments

Experiment 1: savings inflation

We tune each cascade on original MMLU validation. We then report cost at matched large-model accuracy on the MMLU test split versus the subject-matched MMLU-CF split.

We repeat the comparison on reasoning data. There, the contaminated side is original GSM8K and ARC-Challenge, and the fresh side is their dynamically generated counterparts through PromptBench: DyVal problems at complexity levels D1 to D4, and permutation and distractor probes. For causal reasoning, we compare COPA against CausalProbe-2024.

The central number is the savings gap. We predict the naive cascade's savings shrink markedly on fresh data. The gatekeeper's savings should shrink less, and its accuracy should stay near the target.

Experiment 2: mixture sweep

We build evaluation streams that mix contaminated and clean items from 0% to 100% in steps of 20%, using both natural MMLU/MMLU-CF pairs and the controlled WMDP setup. We plot each method's area under the accuracy-versus-cost curve against mixture fraction. Flat curves for our method and sloped curves for fixed thresholds would support claim two.

Experiment 3: asymmetric contamination on WMDP

We compare three conditions: only the small model fine-tuned, only the large model fine-tuned, and both. This tests whether the headroom gatekeeper wastes compute sending questions to a backup whose apparent skill is recall.

Experiment 4: label-free monitoring

On each stream, we compare absolute accuracy estimation error and Poisson-binomial interval coverage for plain average confidence, ATC, the log-likelihood meta-model, and our memorization-conditioned estimator.

Decisive checks

Two checks are decisive:

  • Existence of the calibration gap. On controlled WMDP, reliability curves for contaminated and clean items must differ significantly at equal confidence. If they coincide, the mechanism is absent and the proposal fails at step one.
  • Value of the feature. The gatekeeper with the memorization feature must beat the same gatekeeper without it on the clean test sets, and the random-feature ablation must not.