Selective prediction lets a model decline to answer when it is likely to be wrong, trading coverage (the fraction of questions it answers) for lower risk (the error rate on the questions it does answer). Every published selective-prediction result for language models is measured on public benchmarks such as HotpotQA, MedQA and GSM8K, and these benchmarks have very likely leaked into pretraining data. A model that has memorized a benchmark is confidently right on it for the wrong reason. Its confidence then separates right from wrong answers well on the leaked items, and the abstention threshold tuned there can look safe. We propose to measure how much of reported risk control is an artifact of memorization. We also test a specific repair: a confidence score built from small yes/no checks of whether each reasoning step is supported by the given context, instead of a single score for the whole answer.
The claim in two sentences is this. On contaminated benchmarks, sequence-level confidence ranks correct above incorrect answers partly because it detects familiarity, so both the ranking quality and the chosen threshold degrade on fresh or counterfactual inputs. Binary step checks that ask "is this step stated in or derivable from the passage?" measure grounding rather than familiarity, and should lose much less when the passage contradicts what the model memorized.
Core Claim and Positioning
Two findings motivate the question but have never been put together. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that reinforcement-learning post-training with GRPO (group relative policy optimization) closes the log-probability gap between seen and unseen benchmark questions. After that training, ten contamination detectors fall to 50–58% AUROC while accuracy on the leaked items stays inflated by up to 11.7%. CofCA swaps real entities for counterfactual ones in multi-hop passages and finds that GPT-4's exact match drops from about 70% to about 53% on 2-hop questions. Only 36.3% of its reasoning chains are correct, which means models often reach memorized answers without valid intermediate steps.
The selective-prediction methods we compare against take confidence ranking at face value. Aligning Language Models with Selective Prediction introduces RLSR, which uses reinforcement learning to directly optimize the area under the risk-coverage curve. It shows that ranking matters more than calibration and reports out-of-domain robustness on HotpotQA, GSM8K and MedQA. However, it never separates leaked items from clean ones. ASPIRE learns a self-evaluation prompt that scores whether an answer is valid. It is also sequence-level and is also evaluated only on public QA sets.
The closest structural work is CGRiC and Rethinking LLM Verification. CGRiC breaks structured outputs into binary claim checks to avoid all-or-nothing rejection. Rethinking LLM Verification checks each multiple-choice option with a binary verifier and routes UNKNOWN answers to ontology retrieval. Both supply the decomposition technique we reuse. Neither asks whether their verifiers are fooled by memorization, and the second explicitly leaves MedQA contamination unaddressed.
The binary-decomposition literature tells us why we expect yes/no step checks to be more stable than a graded score. CheckEval raised Krippendorff's alpha (an inter-rater agreement measure) by 0.45 when Likert judging was replaced by binary checklists. Similar gains from removing intermediate classes are reported by Raj et al. and Barrón-Cedeño et al..
What is new here is twofold. The first contribution is a controlled measurement of the "contamination gap" in risk-coverage curves, using known contamination. The second is a premise-anchored binary selector designed specifically to close that gap. This differs from CGRiC and its relatives, which assume static, uncontaminated evaluation and score claims against evidence without testing knowledge conflict.
Research Program
The first component builds ground-truth contamination. Following the protocol in the fragility paper, we take an open-weight 8B-scale model and fine-tune it on a random half of the factual items from CofCA, HotpotQA and a generated DyVal pool. We then apply GRPO so that detector signals are suppressed. This yields three disjoint evaluation populations: seen factual items, unseen factual items, and counterfactual twins of seen items. A counterfactual twin keeps the same question and reasoning structure, but the memorized answer is now wrong. We also add freshly generated DyVal items, which guarantee novelty for the arithmetic and graph tasks.
The second component defines the measurement. For each confidence method we compute the area under the risk-coverage curve, AURC, on each population. We define the contamination gap as AURC on counterfactual twins minus AURC on seen items.
We separate two failure modes. A threshold failure means the selected threshold violates its target risk on new data, while the ranking quality is intact. A ranking failure means AURC itself rises, so no threshold can rescue the method. This distinction matters because only ranking failures require a new method.
The third component is the selector itself. CofCA provides gold sub-questions, and for other datasets we extract them with the model. For each step $k$ of an answer, the model is asked a yes/no question with the passage in context: is step $k$ stated in or derivable from the passage? The response confidence is
$$s(x) = \min_k \; p_k(\text{yes} \mid \text{passage}, \text{step}_k),$$
where $p_k$ is the model's probability of "yes" for step $k$. The minimum reflects the idea that a chain is only as grounded as its weakest step. We also test product aggregation, and we test step-level partial answering in the spirit of CGRiC.
Baselines
The naive reference point is the length-normalized sequence likelihood of the answer, combined with verbalized P(True) prompting. Both are free to compute and both are standard.
The strongest existing method is RLSR. On HotpotQA it reports an AURC of 0.44, against 0.56 for correctness alignment and 0.51 for calibration alignment, along with 66.0% top-decile accuracy against 45–53% for baselines (Aligning Language Models with Selective Prediction). We retrain RLSR on the same contaminated base model, with the same training items and the same GRPO compute as our selector's verifier tuning. We do not import RLSR's published numbers into a different model. We found no reported selective-prediction result on Lacuna for CofCA or DyVal, so all comparisons on those benchmarks are produced inside this study.
ASPIRE serves as the learned sequence-level self-evaluator. Its reported AUROC on CoQA with OPT-2.7B is 80.25%, up from 74.61% (ASPIRE). We tune its soft prompts with the same budget.
Self-consistency, which measures agreement across multiple samples, is run with an equal-compute sample count.
Two ablations isolate the key ingredients. The first removes the premise anchoring: the step check becomes "is this step true?", so it is answerable from memory. The second replaces the binary check with a 1–5 graded support score per step, which tests whether the binary form matters.
Beating the strongest baseline requires two things. Our selector must show a statistically smaller contamination gap than RLSR. It must also show lower AURC on the counterfactual and fresh populations, while losing no more than a small, pre-registered margin on seen items.
Why not just…?
Why not just recalibrate the threshold on fresh, post-cutoff data? If contamination only shifts the threshold, this is the right answer and our selector is unnecessary. We take that possibility seriously. The threshold-versus-ranking decomposition tests it directly. If RLSR's AURC on counterfactual twins matches its AURC on unseen factual items, recalibration suffices and we will report that as the finding. The proposal stands only if ranking itself degrades. CofCA's 36.3% chain accuracy suggests it will, because memorized answers that are confidently produced are exactly the high-ranked errors.
Why not just filter contaminated items upstream with a membership detector? The fragility paper shows that detectors drop to 50–58% AUROC after GRPO. PAC improves black-box detection, but only by 4.5–5.9% AUC on WikiMIA and StackMIA. We include detector-based filtering as an additional baseline on our GRPO-trained model. This means using the detector score as the abstention signal.
Why not just use self-consistency? Memorized answers are reproduced consistently across samples, so agreement should reward the shortcut on counterfactual twins rather than penalize it. CGRiC also reports that self-consistency costs about 400% overhead, against 35–47% for decomposed verification. If self-consistency nevertheless shows a small contamination gap at equal compute, it is a genuine competitor and we will say so.
Why not just use CGRiC's evidence-ablation score? Its score compares the model with and without evidence. A model whose memory agrees with the evidence can still produce lift. CGRiC also needs explicit citation structure. Where HotpotQA provides that structure, we run CGRiC's evidence-ablation score as an additional selector.
Decisive Checks
The central experiment plots risk-coverage curves for every selector on the three CofCA populations and on fresh DyVal items. All selectors run on the same contaminated-then-GRPO model, and we report results across at least three contamination seeds.
The proposal is supported if three conditions hold. First, the sequence-level methods (RLSR, ASPIRE, likelihood) show a clearly larger AURC increase from seen to counterfactual items than the premise-anchored binary selector. Second, the "is this step true?" ablation behaves like the sequence-level methods, which would show that anchoring to the passage, not decomposition alone, does the work. Third, the graded-score ablation shows more variance across paraphrased prompts than the binary version.
As a real-world check, we compare RLSR's target-risk threshold on MedQA against its performance on MMLU-CF and LiveBench medical items. No reported number was found on Lacuna for these benchmarks.
The proposal is refuted if the contamination gap is negligible for RLSR. The modest variant of the result is that recalibration closes the gap, which would still be a useful negative result for the field.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark
- Aligning Language Models with Selective Prediction
- Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
- CGRiC: Compositional Risk Certification for Structured LLM Outputs
- Rethinking LLM Verification: Evidence Structure, Uncertainty, and Selective Refinement
- CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists
- Creating a linguistic plausibility dataset with non-expert annotators
- On the Use of an Intermediate Class in Boolean Crowdsourced Relevance Annotations for Learning to Rank Comments
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Data Contamination Calibration for Black-box LLMs
- MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark