MS-TTRL: Breaking the Frequency Conflict in Multiple-Solution Test-Time Reinforcement Learning

Zhixia Zhang, Binqian Xu, Huaiyang Wang, Yikun Ban, Dianbo LiuarXiv 2026 · arXiv

TL;DR

MS-TTRL (Multiple-Solution Test-Time Reinforcement Learning) resolves the "Frequency Conflict" that plagues models trying to find multiple valid solutions without human labels. By decoupling correctness estimation (via self-verification) from diversity preservation (via frequency-based scaling), it allows models to discover rare but valid reasoning paths, improving solution recovery rates by over 19 percentage points.

What Problem Does This Paper Address?

In many complex reasoning tasks—like scientific discovery or mathematical problem-solving—there isn't just one "right" answer. There might be several valid causal graphs that explain a set of observations or multiple numbers that satisfy a specific property.

Standard Test-Time Reinforcement Learning (TTRL) adapts models to new problems without human labels by using "majority vote" as a proxy for correctness. If the model generates the same answer 80% of the time, TTRL assumes that answer is correct and rewards it.

However, if we want a model to find all valid solutions, we run into the Frequency Conflict:

  1. Correctness Estimation favors high-frequency outcomes (the "safe" majority).
  2. Diversity Preservation favors low-frequency outcomes (the "rare" alternatives).

In a label-free setting, a rare outcome is ambiguous: is it a brilliant, alternative valid solution, or just a rare hallucination? Standard TTRL can't tell the difference, often leading the model to collapse onto a single "easy" solution.

Figure 1: Frequency conflict and signal decoupling in MS-TTRL. Left: Frequency-based cor- rectness estimation favors common outcomes, whereas diversity preservation favors rare ones. Mid- dle: Self-verification estimates correctness, while outcome frequency promotes diversity. Right: MS-TTRL achieves higher Recovery Rate than TTRL and TTRL-CoCoV across four tasks.

Methods

The authors propose MS-TTRL, which breaks this conflict by using three key components:

  • Outcome-Level Self-Verification: Instead of using frequency to guess if an answer is right, the model acts as its own judge. It takes a generated outcome, reasons backward to check if it satisfies the problem's constraints, and calculates a "pass rate." This provides a correctness signal that is independent of how often the answer was generated.
  • Three-State Outcome Cache: To save on the heavy computational cost of self-verification, the system maintains a cache. Outcomes are sorted into three buckets: Correct, Wrong, or Candidate. Once an outcome is resolved as Correct or Wrong, the model reuses that label in future training steps.
  • Confidence-Gated Inverse Probability Scaling (IPS): This is the "secret sauce" for diversity. The system uses Inverse Probability Scaling to give much higher rewards to rare outcomes—but only if they have been verified as "Correct." This prevents the model from accidentally reinforcing rare errors.
Figure 2: Overview of MS-TTRL. For each unlabeled problem, generated solution outcomes are self-verified and maintained in a three-state outcome cache. Their stored status is then combined with gated inverse-probability scaling to reward outcomes, thereby decoupling correctness estima- tion from diversity preservation with limited cost of self-verification.

Datasets

The researchers evaluated MS-TTRL on two major benchmarks:

  1. HypoSpace: A diagnostic benchmark featuring three distinct tasks:
  2. Causal Discovery: Identifying valid directed acyclic graphs (DAGs) from intervention data.
  3. 3D Structure Reconstruction: Inferring 3D voxel shapes from 2D top-down projections.
  4. Boolean Expression Discovery: Finding symbolic expressions that match input-output pairs.
  5. MACE-Math: A mathematical reasoning benchmark where questions are specifically designed to have multiple correct answers (e.g., "Find a prime number between 10 and 50").

Key Insights and Ideas

  • Decoupling is Essential: The "Aha!" moment is realizing that generation frequency is a "double agent." By stripping frequency of its role in correctness estimation and reserving it strictly for diversity reweighting, the conflict disappears.
  • The Power of the "Candidate" State: Not every answer can be verified immediately. By holding uncertain answers in a "Candidate" buffer with soft rewards, the model maintains exploration pressure without prematurely committing to (or rejecting) an outcome.
  • Verification Reuse: Self-verification is expensive. The authors found that a persistent cache can reduce the number of required verification trials by over 75% across the training process, making the method practical for real-world use.
The efficiency of the Three-State Outcome Cache: as training progresses, the number of new samples requiring verification drops significantly.

Key Results

  • Massive Recovery Gains: MS-TTRL improved the Recovery Rate (the percentage of all valid solutions found) by an average of 19.13 percentage points over the best label-free baselines.
  • Closing the Gap with Supervision: Remarkably, MS-TTRL (which uses no labels) approached the performance of IPS-GRPO, a supervised method that has access to ground-truth correctness labels.
  • Robustness: The gains were consistent across different model families (Qwen and Phi) and model sizes (from 1.7B to 8B parameters).
  • Efficiency: Despite the added step of self-verification, the total training time only increased by about 23% compared to vanilla TTRL, thanks to the caching mechanism.

Limitations and Future Work

  • Verifier Quality: The method relies on the model's ability to verify its own answers. If a model is a poor verifier, the training signal will be noisy. The authors suggest that "verification-generation gaps" (where models are better at checking than creating) are helpful here.
  • Task Complexity: While tested on mathematical and logical reasoning, it remains to be seen how this scales to more subjective or creative tasks where "correctness" is harder to define via a reverse-reasoning prompt.
  • Future Directions: The authors point toward exploring more sophisticated verification strategies and applying this decoupling principle to even larger-scale "reasoning" models like DeepSeek-R1.

Summary updated

Abstract

Scientific, engineering, and mathematical reasoning tasks can admit multiple valid solutions, while exhaustive annotation of the full solution set is often prohibitively costly. Test-time reinforcement learning (TTRL) applies reinforcement learning to unlabeled test instances, offering a label-free approach to multiple-solution reasoning. In this setting, generation frequency serves two opposing objectives: correctness estimation favors frequent outcomes, whereas outcome-level diversity preservation favors less frequent outcomes. We term this tension the \textbf{Frequency Conflict}. To resolve this conflict, we propose MS-TTRL, which obtains frequency-independent correctness estimates through Outcome-Level Self-Verification while reserving generation frequency solely for diversity preservation. Three-State Outcome Cache and the Candidate State support verification reuse across repeated observations and confidence-aware handling of ambiguous outcomes, while Confidence-Gated Inverse Probability Scaling selectively strengthens the learning signals of rare, reliably verified outcomes. Experiments across two benchmarks, four multiple-solution tasks, and three backbone models, each evaluated with five random seeds, show that MS-TTRL improves Recovery Rate by 19.13 percentage points on average over the best-performing label-free baseline and approaches the performance of the supervised IPS-GRPO.
BibTeX
@misc{zhang2026ms,
  author = {Zhixia Zhang and Binqian Xu and Huaiyang Wang and Yikun Ban and Dianbo Liu},
  title = {{MS-TTRL: Breaking the Frequency Conflict in Multiple-Solution Test-Time Reinforcement Learning}},
  year = {2026},
  eprint = {2610.32234},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32234},
}
Figure