Equivalence-Class Minimal Pairs for Auditing Contaminated Causal Benchmarks

When a language model scores well on a causal-reasoning benchmark, we cannot tell whether it reasons about the causal graph or remembers the answers. The usual detectors read the model's token probabilities on test items, and they fail on modern reasoning models. After contaminated fine-tuning, one epoch of reinforcement learning on clean data pushes them toward chance, even though the inflated scores remain (On The Fragility of Benchmark Contamination Detection in Reasoning Models). We propose a behavioral audit instead.

For each benchmark item we build "sibling" items using the theory of Markov equivalence classes. A Markov equivalence class is the set of causal graphs that no amount of observational data can tell apart. Some siblings differ from the original by a single reversed arrow yet have a different correct answer. Others are rewritten so heavily that they share little text with the original, yet the correct answer is provably the same. A model that memorized an item tracks how similar the text looks. A model that reasons tracks what the graph implies. Equivalence-class theory lets us vary those two things independently, with proven gold labels.

The opportunity is that causal graphs are one of the few domains where the line between "looks different but means the same" and "looks the same but means something different" can be drawn exactly. Two directed acyclic graphs are equivalent when they share the same undirected skeleton and the same colliders. These are patterns X→Y←Z where X and Z are not connected. Reversing certain arrows keeps a graph inside its class, and reversing others creates or destroys a collider and leaves it. Equivalence-class search methods already move between graphs this way. We repurpose those operations to manufacture contamination probes for which no paraphrase-based or likelihood-based method has a counterpart.

Core Claim

Our claim is that a memorizing model's errors concentrate on one kind of sibling. These are siblings that are textually close to an item it saw in training but whose correct answer differs. A reasoning model's errors should not depend on whether the original item was seen.

Let $s$ be a seed item and $B(s)$ its minimal answer-flipping sibling. We define the contamination statistic as

$$\Delta = \Pr[\hat{a}(B(s)) = a^*(s) \mid s \in \text{seen}] - \Pr[\hat{a}(B(s)) = a^*(s) \mid s \in \text{reference}].$$

Here $\hat{a}$ is the model's answer, $a^*(s)$ is the gold answer to the original seed, "seen" are items suspected to be in training, and "reference" are matched items known to be fresh. A positive $\Delta$ means the model repeats a remembered answer where the graph demands a different one.

Because $\Delta$ is computed from answers rather than token probabilities, we predict it survives the reinforcement-learning stage that erases likelihood gaps. That stage preserves the inflated pass@1 (On The Fragility of Benchmark Contamination Detection in Reasoning Models), which means the memorized answers are still being emitted.

This is different from the closest work. DyVal (DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks) generates fresh graph-based tasks and flips the order in which a graph is described. Order flips are answer-preserving surface changes, and DyVal never asks whether an edit keeps or breaks the equivalence class. CLadder (CLadder: A Benchmark to Assess Causal Reasoning Capabilities of Language Models) and CausalBench (CausalBench: A Comprehensive Benchmark for Evaluating Causal Reasoning Capabilities of Large Language Models) block memorized priors with nonsense or anti-commonsense variable names. Both always hand the model a fully specified graph. CausalProbe-2024 (Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?) relies on post-cutoff news.

None of these produces item-level evidence about whether a specific published item was memorized. None separates textual distance from answer change.

Research Program

The core object is a sibling generator. It takes a causal benchmark item with its underlying graph, query and numbers and emits four siblings, one for each combination of textual distance (small or large) and answer (same or different).

Small edit, same answer. We reverse one edge whose reversal keeps the equivalence class. We pose this sibling as an observational question, such as a conditional-independence query or an observational conditional probability. The text changes by one causal sentence and the answer is unchanged.

Small edit, different answer. We reverse one edge so that a collider is created or destroyed. The edit looks the same size, but the class changes, and so does the answer. We use the same move-set that equivalence-class search uses, including the collider-creating operator of CPDAG-GFN (Learning Equivalence Classes of Bayesian Network Structures with GFlowNet).

Large edit, same answer. We pick a different graph from the same class that is far away in edit distance, then re-verbalize it with new variable names and a new story.

Large edit, different answer. This is a class-changing graph rendered with a heavy rewrite.

Interventional queries add a second axis, and it is the one CLadder-style benchmarks cannot express. Within an equivalence class, the effect of X on Y may differ across member graphs. We therefore add "class-only" siblings in which the model receives only the conditional independencies or a partially oriented graph. The gold answer is then either the identified effect or "not identifiable from the information given."

Gold labels come from established algorithms rather than from the generator's own judgment:

The enumeration counts of Gillispie et al. tell us how many large-edit, same-answer siblings exist at each graph size.

Seeds come from CLadder, whose 10,000 items are produced by a causal inference engine that exposes each item's graph. To add realistic structure, we also seed from the 11-node Sachs protein network and from synthetic Erdős–Rényi graphs of the kind used by DAG-GFlowNet (Bayesian Structure Learning with Generative Flow Networks).

The sibling set is also a contamination-limited benchmark in its own right. However, the scientific question is the audit: does $\Delta$ identify which items a model trained on, in conditions where likelihood detectors cannot?

Baselines

Naive reference. The do-nothing reference is the raw accuracy gap between suspected-seen and fresh items. With ground-truth contamination, this is the very inflation we are trying to attribute. It sets the floor: a detector must separate seen from unseen items at the item level, not merely report that average scores are higher.

Membership-inference detectors. The strongest existing competitor is the suite of likelihood-based detectors evaluated in On The Fragility of Benchmark Contamination Detection in Reasoning Models, including LiRA, Min-K% Prob and loss thresholds. They are the field's standard item-level audit, and the reported numbers are the bar.

That paper reports LiRA reaching 89.1% AUROC on fine-tuning contamination before reinforcement learning. After one epoch of clean GRPO (a reinforcement-learning method), LiRA drops by about 14% and loss by about 16%, and reference-free methods approach 50%. Under chain-of-thought contamination of DeepSeek-R1-Distill-Llama-8B, the ten detectors average 50–58% AUROC despite an 11.7% pass@1 boost.

We run these detectors on exactly the same contaminated checkpoints, items and seen/reference splits as our statistic, using the original paper's configurations, with no extra tuning budget on either side.

Beating them means one of two things. Before reinforcement learning, $\Delta$ must match or exceed LiRA's AUROC. After clean GRPO, and in the chain-of-thought setting, $\Delta$ must stay clearly above the 50–58% band where the likelihood detectors sit.

Answer-preserving perturbation. We also compare against the drop under answer-preserving perturbation in the style of DyVal order flips and EvoEval rewrites. The survey Benchmark Data Contamination of Large Language Models: A Survey reports an average 39.4% drop across 51 models for EvoEval on HumanEval. DyVal reports a 13.6% drop for GPT-4 on reachability when the description order is reversed. We compute the same per-item drop on our large-edit, same-answer siblings and use it as an item-level detector.

Ablation of the key ingredient. The ablation that isolates the proposal's key ingredient replaces our graph-derived small-edit, different-answer siblings with generic counterfactual edits that change a number or a stated probability. This keeps the "small edit, flipped answer" design but drops equivalence-class structure. If the generic version matches ours, the graph theory is buying label correctness and the large-edit, same-answer arm, not detection power. We would report that plainly.

Why not just…?

Why not just use Min-K% or LiRA? Because the evidence says they stop working exactly where auditing matters most. The fragility study shows clean reinforcement learning shrinks the log-probability gap between seen and unseen items while keeping the inflated scores (On The Fragility of Benchmark Contamination Detection in Reasoning Models). Our experiment reproduces that setting and measures both detectors side by side, so this objection is settled empirically rather than argued.

Why not just watermark benchmarks before release? Watermarking works well on question text. Detecting Benchmark Contamination Through Watermarking reports p-values of $10^{-3}$ at a +5% ARC-Easy gain and below $10^{-12}$ after 16 exposures. But it must be applied before publication, so it cannot audit CLadder or any benchmark already on the web. The authors also note that short answers carry too little signal and that paraphrasing can wash the mark. Our audit is retroactive and needs no secret key. The two approaches are complementary rather than competing.

Why not just paraphrase items or generate fresh ones? Answer-preserving paraphrase measures fragility, not memorization. A model can fail paraphrases because it is brittle, and the chain-of-thought contamination results suggest contamination can also lift paraphrases. Fresh generation, as in DyVal or post-cutoff collection as in CausalProbe-2024, avoids contamination but says nothing about which published items were seen, and post-cutoff sets decay once they are scraped.

Why not just use generic minimal edits? This is the real threat. Generic answer-flipping edits might capture most of the signal. Our ablation tests this directly. Where graph theory is irreplaceable is the large-edit, same-answer arm and the identifiability siblings, where no surface rewrite can guarantee the gold label.

Proposed Experiments

Controlled contamination. Following the fragility study's protocol, we contaminate Qwen2.5-7B-Instruct by supervised fine-tuning on half of a CLadder-derived seed pool. We then run one epoch of clean GRPO. Separately, we contaminate DeepSeek-R1-Distill-Llama-8B with chain-of-thought solutions. At each checkpoint we compute the AUROC of $\Delta$ for separating seen from reference seeds, alongside every baseline above.

Decisive check. The decisive check is the post-reinforcement-learning checkpoint. If $\Delta$'s AUROC stays well above the 50–58% band while LiRA collapses, the core claim holds. If $\Delta$ also collapses, then reinforcement learning is changing answers and not just likelihoods, and the claim fails.

Diagnostic pattern in the four cells. A second check reads the error pattern across the four sibling cells. Memorization predicts errors concentrated in the small-edit, different-answer cell for seen items only. A memorizer should also stay accurate on the small-edit, same-answer cell, despite the arrow reversal, because the text barely changes.

Chain-of-thought regime. The chain-of-thought regime tests a harder case. If contamination there installs a transferable shortcut rather than item recall, $\Delta$ may be near zero while accuracy on class-only identifiability siblings drops for all items. That pattern would itself tell us what the contamination installed.

Audit of open models. Finally, we apply the audit to open models on public CLadder items and on fresh siblings. As a sanity reference we report GPT-4 with CausalCoT at 70.40% on CLadder. We found no reported result on Lacuna for any model on equivalence-class siblings or identifiability queries.