Invariant Consensus: Pseudo-Labeling When the Whole Committee Has Seen the Test

A common way to label data without humans is to ask several different models and keep only the answers they agree on. This is consensus pseudo-labeling: the models' shared answer becomes a surrogate training label. It rests on one bet, that independent models rarely make the same mistake. Modern language models break that bet in a specific way. They are trained on overlapping web crawls that contain leaked benchmark questions, answer keys and reasoning templates. So they can agree because they memorized the same text, not because each worked the problem out. When that happens, the agreement filter lets in the answers it was built to reject, and adding more models from different families mostly adds copies of the same memorized signal.

We propose to test, under controlled conditions, whether this "contaminated consensus" is real and large. We also propose a simple fix. A pseudo-label is accepted only if the committee's agreement survives answer-preserving rewrites of the question: reordered premises, renamed entities, re-rendered surface forms. Memorized agreement is tied to a particular wording and should break under rewording. Agreement that comes from actually solving the problem should not. We measure this on problem generators whose ground truth is known, and we plant contamination deliberately so we know exactly which agreements are fake.

Core Claim

Our claim has three parts. First, shared exposure to leaked data raises the error correlation among nominally different models on exactly the items where consensus filtering is most confident. This makes raw agreement a biased signal of label quality. Second, it makes committees saturate earlier: past two or three members, extra models add almost no independent evidence. Third, requiring agreement to hold across rewrites that preserve the answer removes most of this bias. It does so without contamination detectors, which the literature shows can be silenced by later training.

The saturation part has a simple quantitative form. Suppose $K$ members have average pairwise error correlation $\rho$. The standard variance calculation for an average of correlated variables gives an effective number of independent voters:

$$K_{\text{eff}} = \frac{K}{1 + (K-1)\rho}.$$

This caps near $1/\rho$ however large $K$ grows. We predict that $\rho$, measured separately on leaked and clean items, is much larger on leaked items for raw agreement. We further predict it falls back toward its clean-item value when agreement is computed over rewrites.

Where This Sits Relative to Prior Work

Four groups of papers matter here, each in a different role.

The first group motivates the problem by showing that agreement-based filtering works when errors are independent.

The first two papers name shared teacher bias as an open limitation but never measure it. The third works in closed-set segmentation, where leaked pretraining data is not the issue.

The second group supplies the saturation lens, and one paper is a direct competitor.

None of these asks where the redundancy comes from. The Ising approach in particular learns one coupling per pair of judges from their outputs. Contamination-driven correlation is different: it is item-specific, strong on leaked items and absent on fresh ones. A global coupling blurs these together. Our method estimates correlation per item, by rewriting that item.

The third group constrains the design. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that a membership detector, LiRA, catches supervised fine-tuning contamination at 89.1% AUROC. After one epoch of GRPO, a reinforcement-learning method, loss-based detectors drift toward about 50%. Detectors fail almost entirely under chain-of-thought contamination. This rules out "run a detector, then drop flagged items" as the main fix, and it tells us to include a reinforcement-learning-cloaked condition.

Complementing Self-Consistency with Cross-Model Disagreement uses a jury of 7–9B models from different families to estimate epistemic uncertainty, meaning uncertainty from what a model does not know rather than inherent noise. It gains up to +0.15 AUROC on HotpotQA. The authors themselves note that this signal can vanish if the models share web text. We test that directly.

The fourth group supplies the testbed.

  • DyVal generates reasoning problems from directed acyclic graphs at four difficulty levels. Its answers are known, and it can render the same graph many ways. GPT-4 loses 13.6% on reachability when the description order is reversed, which is exactly the brittleness our filter relies on.
  • Can Large Language Models Reason? A Characterization via 3-SAT gives a hardness dial for logic problems. GPT-4 drops to roughly 10% at the phase transition, where shortcut heuristics fail.

Research Program

The committee. We start from a pool of open-weight 7–9B instruction models from distinct lineages: Llama, Mistral, Qwen, Gemma and Granite, the families used by the cross-model uncertainty work. Committee size $K$ runs from 1 to 5.

Planted contamination. We generate a DyVal pool (arithmetic, Boolean logic, deductive logic, reachability, levels D1–D4) and a 3-SAT pool spanning the hardness range. Part of each pool becomes a leak set. We fine-tune $c$ of the $K$ members on the leak set with two kinds of answer attached.

  • Correct-answer leakage mimics ordinary benchmark leakage.
  • Planted-wrong-answer leakage attaches a fixed wrong answer to each leaked item. Any committee agreement on that answer is contamination-driven by construction.

We vary $c$ from 0 to $K$ to control how widely the exposure is shared.

In a third condition, contaminated members then receive one epoch of GRPO on unrelated math data. This reproduces the cloaking effect from the fragility paper. A fourth, more realistic condition leaks chain-of-thought traces rather than bare answers.

Invariant consensus. For each unlabeled item, a rewriter produces $V$ variants whose answer is guaranteed unchanged:

  • premise order is shuffled;
  • entity and variable names are changed;
  • numbers are re-expressed without changing their values;
  • for DyVal, the same graph is re-rendered from its generator.

Because the generators are programmatic, answer preservation is checked, not assumed. Each member answers every variant.

We compute two agreement scores. Raw agreement is computed on the original item only. Invariant agreement is the fraction of variants on which a single answer is the plurality answer across members. We then fit the correlation $\rho$ per item class (leaked vs. fresh) under both scores. We trace pseudo-label precision and coverage as $K$ grows, which gives a saturation curve.

Student training. A student model, smaller than the committee members, is fine-tuned on the accepted pseudo-labels from a fresh unlabeled pool mixed with leaked items. It is evaluated on newly generated DyVal and 3-SAT instances that no member has seen. It is also evaluated on the math and reasoning categories of LiveBench, for transfer to problems written after the models' training cutoffs.

Natural-data follow-up. We repeat the analysis on GSM8K and HotpotQA, where leakage is suspected but not planted. We use the same kinds of rewrites, and paraphrases are verified by checking that the answer is recomputed and unchanged. The goal is to see whether the leaked-versus-fresh correlation gap we observe under planted contamination also shows up as a raw-versus-invariant gap in the wild.

Baselines

Naive reference points. One is unfiltered pseudo-labeling: the student trains on every answer from the single strongest member. The other is single-model confidence thresholding. The dual-teacher MT paper shows unfiltered synthetic data can fall below the no-synthetic-data baseline (41.8 vs. 42.7 chrF++). We expect the same pattern, and we count any method that fails to beat these two as useless.

Raw-agreement consensus. This is plain majority vote over $K$ heterogeneous members with a unanimity or supermajority threshold, in the style of the Dual-Teacher Agreement and tri-training work (Asymmetric Tri-training). It is the method we claim contamination breaks. It gets the same total compute as ours, meaning $K \times V$ forward passes, spent as extra temperature samples on the original wording.

Dependence-aware Ising aggregation. This is the strongest competitor. It is the class-dependent Ising model with EM fitting from Dependence-Aware Label Aggregation, which reports 0.912 accuracy on WikiQA against 0.820 for conditional-independence weighted voting. It is the strongest because it directly models the correlation we care about. We found no reported result on Lacuna for Ising aggregation on DyVal, 3-SAT or any contamination-controlled setting, so we run it ourselves. It gets the same members and the same $K \times V$ outputs, pooled so it can use the variants as extra observations. We tune its hyperparameters on the same held-out split with the same number of trials as ours.

Cross-model epistemic-uncertainty filtering. This keeps items with low total uncertainty as defined in Complementing Self-Consistency with Cross-Model Disagreement, at the same retention rate as ours.

Detector-then-filter. This drops items flagged by Min-K% or LiRA before voting. We expect it to work only in the uncloaked condition.

Ablations isolating the key ingredient. There are two. One is invariance without heterogeneity: a single model answering all $V$ variants, several times each. The other is heterogeneity without invariance, which is the raw-agreement baseline.

What counts as winning. Beating the Ising baseline means two things. At matched coverage, pseudo-label precision on leaked items must be at least 5 points higher. Student accuracy on freshly generated items must be higher, with non-overlapping confidence intervals over three seeds. The margin must hold in the GRPO-cloaked condition, not only the simple one.

Why not just…?

Why not just use one frontier model instead of a committee? Single strong models fail precisely on the hard, clean items where labels matter most. GPT-4 falls to about 10% at the 3-SAT phase transition (3-SAT), and o1-preview reaches only 64.7% on LiveBench. A single model also has no second opinion against its own memorized errors. We still include the strongest available member alone as a labeler, so this claim is tested rather than assumed.

Why not just fit the Ising dependence model and let it discount correlated judges? This is a real threat. If contamination-driven correlation were roughly constant across items, the Ising model would absorb it. Our bet is that the correlation is concentrated on the leaked subset. A single coupling per pair then either underweights honest agreement on fresh items or overweights fake agreement on leaked ones. The planted-wrong-answer condition settles this directly. If the Ising model rejects planted answers as well as invariant consensus does, our method is unnecessary. In that case we would report that, and recast the contribution as the per-item correlation measurements.

Why not just ask one model the rewritten variants? This may be the strongest alternative. Rewording alone might break memorization, making heterogeneity redundant. Two considerations cut against it. Shortcut heuristics that survive rewording, such as length cues in 3-SAT, are model-specific. And self-consistency within one model is known to collapse when the model is confidently wrong (Complementing Self-Consistency; ReConcile). The single-model-invariance ablation decides this. If it matches the full method at equal compute, we will say so and recommend the cheaper version.

Why not just detect contaminated items and remove them? The fragility results show that loss-based detectors lose most of their power after a single epoch of GRPO, and fall to near chance under chain-of-thought leakage (On The Fragility of Benchmark Contamination Detection). Mechanistic detectors such as RADAR need access to model internals and were validated on 100 instances. Our cloaked condition tests detector-then-filter head to head.

Decisive Checks

Four results would decide the claims.

Leaked-versus-fresh correlation gap. Under raw agreement, measured error correlation $\rho$ should be clearly higher on leaked than on fresh items. With planted wrong answers, raw agreement should accept those answers at a rate growing with $c$. If neither holds, the premise fails and the project stops there.

Earlier saturation. Raw-consensus precision should plateau at smaller $K$ as $c$ grows, while invariant-consensus curves keep improving. $K_{\text{eff}}$ should be computed both ways to show where the plateau comes from.

Cloaking condition. After GRPO, detector-based filtering should fall toward chance, while invariant consensus keeps its advantage over both raw voting and the Ising aggregator.

Downstream value. Students trained on invariant-consensus labels should score higher on freshly generated DyVal D3–D4 and near-transition 3-SAT instances, and on LiveBench math and reasoning. If gains appear only in label precision and not in the student, the filter is removing easy items rather than fake ones. A per-difficulty breakdown of retained items will show which.