Abstract
Test-Time Reinforcement Learning (TTRL) has emerged as a powerful technique for improving the reasoning capabilities of Large Language Models (LLMs) by using model-generated samples as pseudo-labels. However, standard TTRL relies on majority voting, which is prone to confirmation bias and provides sparse reward signals. We introduce SCOPE (Subgroup-specific step-wise COnfidence-weighted Pseudo-label Estimation), a framework that replaces simple majority voting with step-wise confidence weighting and utilizes subgroup partitioning to provide denser, more reliable rewards. SCOPE achieves significant performance gains across rigorous mathematical benchmarks, including a 51.1% relative improvement on AIME 2025 for lightweight models.
1. Introduction
Reinforcement Learning with Verifiable Rewards (RLVR) has been instrumental in the success of models like DeepSeek-R1. However, RLVR requires external verifiers (e.g., compilers or ground-truth labels) which are often unavailable. TTRL addresses this by using the model's own consensus as a reward signal.
As illustrated in Figure 1, standard TTRL faces two primary challenges: Confirmation Bias, where popular but incorrect answers are reinforced, and Sparse Rewards, where a single global consensus fails to capture the diversity of valid reasoning paths. SCOPE mitigates these by integrating internal model confidence and local subgroup consensus into the reward calculation.
2. The SCOPE Framework
SCOPE introduces three key innovations to refine the reward signal:
2.1 Step-wise Confidence Weighting
Rather than treating all samples equally, SCOPE calculates an Average Step Confidence based on the model's internal log-probabilities at each reasoning step. This allows the framework to prioritize answers where the model exhibited consistent certainty throughout the logic chain, effectively filtering out "hallucination traps" that might otherwise win a simple majority vote.
2.2 Subgroup Partitioning
To provide denser feedback, SCOPE divides the sample pool into smaller subgroups. Each subgroup derives a "local consensus" via bootstrap sampling. This approach provides the model with multiple supervision targets, encouraging the exploration of diverse reasoning paths and preventing mode collapse toward a single global average.
2.3 Pareto Optimization for Subgroup Size
The optimal subgroup size ($m$) is determined dynamically using Pareto optimization. This process balances Reasoning Quality (the accuracy of the local consensus) against Exploration Diversity (the variety of reasoning paths). By minimizing a trade-off distance function, SCOPE automatically selects the most effective granularity for the reward signal.
3. Experimental Evaluation
SCOPE was evaluated on several high-stakes mathematical reasoning benchmarks, including AIME 2024/2025, AMC, and MATH-500. The framework was tested across multiple model families, ranging from lightweight (Qwen2.5-Math-1.5B) to medium-sized (Qwen3-8B and Llama-3.1-8B) architectures.
4. Results and Analysis
Experimental results demonstrate that SCOPE consistently outperforms standard TTRL and other unsupervised RL baselines across all tested benchmarks and model sizes.
As shown in Table 1, the improvements are particularly pronounced for smaller models. For instance, Qwen2.5-Math-1.5B saw a 51.1% relative performance increase on AIME 2025. Even for larger models like Qwen3-8B, SCOPE achieved a significant 13.1% improvement on the same benchmark. Qualitatively, models trained with SCOPE were observed to produce more elegant mathematical derivations, avoiding the logical "hacks" sometimes seen in standard TTRL training.
5. Conclusion
SCOPE represents a significant step forward in making Test-Time Reinforcement Learning more reliable and effective. By moving beyond the "ballot box" of majority voting and incorporating fine-grained confidence and subgroup diversity, the framework provides a more robust signal for model alignment. Future work will explore scaling this approach to longer-horizon tasks such as complex coding and interactive agent environments.
Summary updated