On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning

Haoyu Wang, Ru Wang, Yuanyuan Wang, Jiaxian Guo, Mingming GongarXiv 2026 · arXiv

TL;DR

This paper introduces PC-Dr.DPO, a robust framework for test-time reinforcement learning (TTRL) that addresses the inherent unreliability of self-generated "majority-vote" labels. By dynamically adjusting the strength of model updates based on the stability of its own votes and employing a cache-based safeguard, the authors significantly improve adaptation performance across 30 different model-benchmark combinations.

What Problem Does This Paper Address?

Large Language Models (LLMs) are typically trained offline, meaning they can't easily adapt to new, specialized domains once deployed. Test-time reinforcement learning (TTRL) aims to fix this by letting a model learn from its own outputs on unlabeled data.

A common strategy is "majority voting": the model generates several answers to a prompt, and the most frequent answer is treated as the "ground truth" (a pseudo-label). However, this is risky. The authors identify two main failure modes:

  1. Population-majority bias: The model is fundamentally wrong, and its most likely answer is incorrect.
  2. Finite-rollout variation: Because we only sample a few answers (e.g., 32), the "majority" might just be a lucky (or unlucky) fluke of sampling.

Existing methods often treat all pseudo-labels as equally valid, which can lead to "training collapse" where the model becomes confidently wrong.

Figure 1: Distribution of prompt-level modal-answer agree- ment for Qwen2.5-Math-1.5B-Instruct on the 83-problem AMC benchmark. For each prompt x, we sample m = 32 rollouts and compute the fraction producing the most fre- quent final answer. The broad distribution indicates substan- tial prompt-level heterogeneity: values near one correspond to nearly unanimous rollout groups, whereas smaller values indicate fragmented answer distributions that are more sen- sitive to resampling.

Methods

The authors propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO). The method features three key innovations:

  • Prompt-Conditional Robustness: Unlike standard robust optimization that applies a blanket "noise filter" to a whole dataset, PC-Dr.DPO looks at each prompt individually. If the model's answers for a specific prompt are split (low agreement), the algorithm performs a more conservative update. If the answers are nearly unanimous, it updates more aggressively.
  • Reliability-Conditioned Temperature: They use the "modal-answer fraction" (how many samples agreed with the majority) as a proxy for reliability. This value is used to set a prompt-specific "temperature" that scales the gradient updates.
  • Cache-Based Stabilization: To prevent "self-forgetting"—where a model loses a correct reasoning path it previously found—they implement a cache. If a prompt becomes "degenerate" (all samples produce the same answer), that answer is cached. If future samples disagree with this stable history, the model reverts to the cached version to anchor its learning.

Datasets

The authors evaluated their method across six challenging benchmarks using five different model backbones (Qwen2.5 and Llama-3.2 families):

  • Mathematical Reasoning: AIME24, AIME25, AMC23, and MATH500.
  • Knowledge-Intensive Reasoning: GPQA-Diamond (graduate-level Q&A) and MMLU-Pro (mathematics subset).

Key Insights and Ideas

  • Sampling vs. Bias: Increasing the number of samples ($m$) helps reduce "sampling luck" (finite-rollout variation), but it does nothing to fix "population bias." If a model is 90% sure of a wrong answer, sampling more will only make it more certain of that error.
  • The "Aha" Moment on Heterogeneity: As shown in Figure 1, some prompts are "easy" for the model to agree on, while others are "noisy." A one-size-fits-all update rule is mathematically suboptimal for test-time adaptation.
  • Self-Reinforcing Collapse: The paper provides a stark visualization of "collapse" (Figure 4). When a model repeatedly optimizes against incorrect but decisive pseudo-labels, its internal consistency goes up while its actual accuracy drops to zero.
Figure 2: Adaptation dynamics of Qwen2.5-Math-1.5B-Instruct on the AMC benchmark. Left: fraction of problems in the initial majority-vote-solvable set that are recovered by the adapted policy. Middle: difference between the number of problems solved by the current policy and the initial majority-vote reference; the dashed line at zero denotes full recovery. Right: decomposition of the currently solved problems into those recovered from the initial majority-vote-solvable set and newly discovered solutions. The policy progressively recovers most initially available solutions while occasionally solving problems outside the initial set.

Key Results

PC-Dr.DPO matched or outperformed the strongest TTRL baselines in 24 out of 30 experimental settings.

  • Quantitative Gains: On the MATH500 benchmark with a Qwen2.5-3B backbone, PC-Dr.DPO achieved 73.3% accuracy, compared to 63.5% for the base model and 70.9% for the standard TTRL baseline.
  • Stability: The method was significantly more resistant to the "training collapse" that plagued other methods like RENT or standard DPO on difficult benchmarks like AIME.
  • Recovery: Qualitative analysis (Figure 2) shows the model progressively "recovers" solutions it could initially only find through lucky sampling, turning them into stable, high-probability behaviors.
Figure 2: Adaptation dynamics showing how the model progressively recovers solutions from its initial "lucky" majority votes and even discovers new solutions.

Limitations and Future Work

  • Unidentifiable Bias: The authors admit that "population-majority bias" is generally unidentifiable without external supervision. If the model is consistently wrong across its entire distribution for a prompt, PC-Dr.DPO cannot fix it; it can only try to mitigate the damage.
  • Computational Cost: Generating 32 rollouts per prompt for every update step is computationally expensive, which might limit real-time deployment in resource-constrained environments.
  • Future Directions: The authors suggest that future work could focus on better ways to detect population bias without ground truth, perhaps by looking at the "reasoning traces" (Chain-of-Thought) rather than just the final answer.

Summary updated

Abstract

Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose \emph{Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO)}, which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. We further introduce a cache-based mechanism to reduce self-supervised forgetting during online adaptation. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.
BibTeX
@misc{wang2026reliability,
  author = {Haoyu Wang and Ru Wang and Yuanyuan Wang and Jiaxian Guo and Mingming Gong},
  title = {{On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning}},
  year = {2026},
  eprint = {2611.09872},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09872},
}
Figure