Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning

Sitong Wu, Haoru Tan, Xichen Zhang, Bin Xia, Shaofeng Zhang, Xiaojuan Qi, Bei Yu, Jiaya JiaICML 2026 · OpenReview

Test-Time Reinforcement Learning (TTRL) has emerged as a powerful technique to improve Large Language Model (LLM) performance at inference time without requiring ground-truth labels. Traditionally, TTRL relies on a "wisdom of the crowd" approach known as majority voting: the model generates multiple solution trajectories, and the most frequent answer is assumed to be correct, serving as a "pseudo-label" for reinforcement learning.

However, this assumption frequently breaks down in complex reasoning tasks, such as competition-level mathematics. Models often fall into "logical traps" or produce a multimodal distribution of errors, where several different incorrect answers collectively outvote the single correct one. In these scenarios, the correct solution resides in the logical minority. Standard TTRL hits a performance ceiling because it systematically undervalues and discards these rare, correct trajectories.

To address this fundamental flaw, researchers have introduced Self-Reflective Test-Time Reinforcement Learning (SR-TTRL), a novel framework that replaces statistical consensus with logical verification.

The SR-TTRL Framework

SR-TTRL leverages the model's own reflective capabilities to identify logically sound solutions through a three-step pipeline:

  1. Candidate Trajectory Pool Construction: The model generates $N$ trajectories, which are then grouped by their final answers. One representative trajectory is selected from each group. This ensures the model evaluates a diverse set of unique solutions, removing the bias of frequency.
  2. Trajectory Summarization: Reasoning paths can be thousands of tokens long and filled with stylistic noise. To prevent "context overflow" during evaluation, the model is prompted to condense these paths into "reasoning skeletons"—concise summaries that preserve only the core mathematical derivations and logical transitions.
  3. Self-Reflective Verification: Acting as its own judge, the model reviews the summarized candidate pool side-by-side. It performs a cross-comparison of the logic and selects the most plausible reasoning path. This chosen answer becomes the high-fidelity pseudo-label used to calculate rewards for the RL update.

Key Insights and Performance

Evaluated across challenging mathematical reasoning benchmarks (AIME24, AIME25, AMC23, and MATH-500) using Qwen3 and LLaMA 3.1/3.2 model families, SR-TTRL demonstrated significant advantages over traditional methods.

  • Massive Accuracy Gains: On the AIME24 benchmark, SR-TTRL boosted the Qwen3-8B model's Pass@1 accuracy from 29.1% to 55.8% (a +26.7% gain), outperforming the standard TTRL baseline by an additional 9.1%. Similarly, LLaMA3.2-3B-Instruct saw a five-fold improvement on AIME24, jumping from 3.8% to 20.0%.
  • Solving the Scaling Paradox: The authors observed that simply increasing the number of samples ($N$) in standard TTRL often fails to improve the majority vote (Maj@N), as more samples just amplify the noise of logical traps. By identifying correct solutions in the logical minority, SR-TTRL breaks this ceiling, allowing the model to learn from its own "lucky" successes.
  • Unprecedented Sample Efficiency: SR-TTRL prioritizes quality over quantity. The model achieves better results with just 8 samples using self-reflection than standard TTRL achieves with 64 samples using majority voting.
  • Narrowing the Oracle Gap: When compared to an "Oracle" baseline (training with actual ground-truth labels), SR-TTRL recovered nearly half of the lost potential that majority-voting methods failed to capture.

Limitations and Future Directions

While highly effective, SR-TTRL comes with trade-offs. The additional steps of summarization and self-reflection add approximately 28% more latency per training step compared to simple majority voting. Furthermore, the method fundamentally relies on the base model's inherent ability to recognize a correct path; if a model lacks the capacity for self-reflection, the generated pseudo-labels will remain low-quality.

Currently, the framework has been validated exclusively on mathematical reasoning. Future work will need to explore its efficacy in Agentic (multi-step tool use) and Multimodal environments, potentially broadening the scope of self-reflective learning across diverse AI domains.

Summary updated

Abstract

The core challenge of Test-Time Reinforcement Learning (TTRL) lies in estimating rewards without access to ground-truth supervision. Existing TTRL methods predominantly rely on majority voting to generate pseudo-labels, under the assumption that the most frequent answer among sampled trajectories is correct. However, we observe that this assumption frequently breaks down in complex reasoning tasks, where correct solutions often constitute a logical minority. As a result, rare yet correct trajectories are systematically undervalued by majority-voting-based approaches. To address this limitation, we propose Self-Reflective Test-Time Reinforcement Learning (SR-TTRL), a novel framework that leverages self-reflective verification to produce high-fidelity pseudo-labels. Specifically, given multiple sampled trajectories for a problem, SR-TTRL first groups trajectories according to their final answers and selects one representative from each group to form a candidate pool. Each candidate trajectory is then summarized to preserve its core reasoning steps while reducing verbosity. Finally, the model performs self-reflection over the candidate pool, critically evaluating and selecting the most plausible trajectory as the pseudo-label. Empirically, SR-TTRL achieves substantially higher pseudo-label fidelity and sample efficiency than prior majority-voting-based TTRL methods. Extensive experiments across diverse benchmarks and model families demonstrate that SR-TTRL consistently outperforms majority-voting baselines and significantly improves generalization to novel problems. For example, SR-TTRL improves the Pass@1 accuracy of Qwen3-8B on AIME24 from 29.1 to 55.8 (a gain of +26.7), exceeding standard TTRL by an additional +9.1. The code will be released at: https://github.com/JIA-Lab-research/SR-TTRL.
BibTeX
@inproceedings{wu2026beyond,
  author = {Sitong Wu and Haoru Tan and Xichen Zhang and Bin Xia and Shaofeng Zhang and Xiaojuan Qi and Bei Yu and Jiaya Jia},
  title = {{Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=kSuRY5sCuR},
}
Figure