When Does Exploration Prevent Collapse? Three Regimes and the Path of the Reward in Self-Consistency Training

Jaeho Hwang, Uiwon HwangarXiv 2026 · arXiv

TL;DR

Self-consistency training (TTRL) often causes models to "collapse" onto their initial majority answers, even when they are wrong. This paper provides a theoretical framework to calculate the exact exploration strength needed to prevent this collapse, revealing that placing exploration bonuses before normalization in algorithms like GRPO is critical for stability and maintaining answer diversity.

What Problem Does This Paper Address?

Large Language Models (LLMs) can often improve their own reasoning by sampling multiple answers and rewarding those that agree with the majority—a technique known as Test-Time Reinforcement Learning (TTRL). Because it doesn't require ground-truth labels or external verifiers, it’s a powerful tool for on-the-fly adaptation.

However, TTRL has a "rich-get-richer" problem: the reward signal favors what the policy already prefers. This leads to policy collapse, where the model's probability distribution narrows down to a single answer. If the initial majority was incorrect, the correct answer can be permanently pruned from the model's samples. While researchers use "exploration" (like entropy bonuses or temperature scaling) to fight this, the required strength of these bonuses is usually found through tedious trial and error. This paper seeks to turn that trial-and-error into a science.

Methods

The authors model the training process as a continuous flow of logits (the "tabular flow") for a softmax policy. They analyze the stability of different policy states (collapsed vs. broad) as the exploration coefficient varies.

Key methodological contributions include:

  • Stability Analysis: They identify three distinct regimes of behavior based on the exploration strength.
  • The "Path of the Reward": They track how the reward signal travels through the policy gradient, the optimizer (like Adam), and the normalization step (like the standard deviation division in GRPO).
  • Closed-Form Boundaries: For common rewards (like agreement probability) and exploration terms (like smoothing toward a uniform policy), they derive exact mathematical boundaries for when collapse is inevitable.
  • Computer-Assisted Proofs: For complex scenarios like Vision-Language-Action (VLA) models that use ordered bins, they use interval arithmetic to rigorously certify stability boundaries.

Datasets

The theory was validated across three increasingly realistic settings:

  1. Network Testbed: A synthetic setup with 32 states and 256 ordered action bins.
  2. Letter Test: Qwen2.5-1.5B-Instruct answering multiple-choice questions from MMLU-Pro, where the model's internal distribution over letters was computed exactly.
  3. Free-form TTRL: Qwen2.5-Math-1.5B solving 128 problems from the MATH-500 dataset.

Key Insights and Ideas

  • The Three Regimes: Exploration isn't a simple "more is better" dial. There is a lower boundary (below which only collapsed policies are stable) and an upper boundary (above which only broad, diverse policies are stable). In the "window" between them, the model's final state depends entirely on its starting point—a phenomenon known as hysteresis.
Figure 1. Regimes in four settings, with no window in (c). Lines are stable (solid) and unstable (dashed) equilibria of the tabular flow, with shading where collapse cannot be prevented (dark) and in the window (light). Markers give the mean pmax of a run from broad (squares) and collapsed (circles) starts, filled and open for SGD and AdamW in (a) and (c) and for seeds 0 and 1 otherwise. (a) Testbed, Gaussian kernel of width 3 in the loss, with the folds of Theorem 4.6 times 62
  • Placement is Everything: A central "aha!" moment is that the effectiveness of an exploration bonus depends on where it enters the "path of the reward." If you add a bonus before the GRPO normalization (Arm C), the threshold for preventing collapse is a simple constant ($1/\log(G-1)$ for $G$ samples). If you add it after (Arm R), the threshold becomes 2-3x higher and depends on the model's current policy, making it much harder to tune.
  • Exploration Preserves, but Doesn't Correct: The authors prove that symmetric exploration (treating all answers equally) preserves the order of probabilities. This means exploration can keep a correct minority answer from disappearing, but it cannot magically turn a wrong majority into a right one. You still need a "symmetry-breaking" signal (like a verifier) for that.
  • Early Prevention is Cheap: The coefficient needed to prevent a model from committing to a leader is lowest when the model is still uncertain. Once a leader holds 50% of the probability mass, the cost to stop the collapse rises significantly.

Key Results

  • Theoretical Thresholds: For a group of $G=8$ samples, the predicted bonus threshold before normalization was 0.514. In the MATH-500 experiments, a bonus of 0.6 successfully prevented collapse across all seeds, while a bonus of 0.45 did not.
  • Diversity vs. Accuracy: In the free-form math task, adding the exploration bonus before normalization (Arm C) kept more correct answers in the sample pool. This raised the pass@32 score to 0.710 (compared to 0.561 for the control). However, it lowered pass@1, because the model was no longer "betting the farm" on its top choice.
  • Normalization Shift: The experiments confirmed that GRPO's normalization "splits" a single threshold into two, creating a wider window where a model might stay collapsed even if the bonus is technically strong enough to have prevented it initially.

Limitations and Future Work

  • Single-State Theory: The current theory treats each problem state in isolation. In practice, LLMs share parameters across many questions, which couples their collapse dynamics.
  • Mean-Field Normalization: The analysis of GRPO uses a mean-field model for the standard deviation, which might differ slightly from small-batch empirical statistics.
  • Symmetry Breaking: Since symmetric exploration can't fix a wrong majority, future work is needed to combine these stability results with verifiers or other signals that can distinguish between "good" and "bad" consistency.
  • VLA Models: While the paper provides tools for ordered bins, full-scale testing on robotic VLA models remains an open area.

For more details, see the full paper: "When Does Exploration Prevent Collapse? Three Regimes and the Path of the Reward in Self-Consistency Training" (Hwang & Hwang, 2026).

Summary updated

Abstract

Test-time reinforcement learning (TTRL) adapts a model without labels by rewarding answers that agree with the majority, but it collapses the policy onto its majority answer and can remove the correct answer from the samples. Exploration can prevent this collapse, yet its strength is chosen by trial and error, and how the training algorithm changes the required strength is unclear. We model training on a single problem as a continuous flow of the logits of a softmax policy and analyze which policies are stable as the exploration coefficient varies. Only collapsed policies are stable below a lower boundary and only policies spread over many answers above an upper boundary, and we derive these boundaries in closed form for an agreement reward with smoothing toward the uniform policy, compute them for the majority reward of TTRL on policies with one leading answer and bound them with interval arithmetic for actions on ordered bins. We prove that under exact gradients, the learning rate and, in a mean-field model, the normalization of GRPO leave these boundaries unchanged when exploration passes through them together with the reward but shift them when exploration bypasses them. Exact sums on these policies then show that a bonus for rare answers needs a coefficient above $1/\log(G-1)$ for groups of $G$ samples before the normalization and about two to three times that after it. In free-form TTRL on MATH-500, a bonus placed before the normalization prevented collapse and kept more correct answers among the samples, although it lowered pass@1.
BibTeX
@misc{hwang2026when,
  author = {Jaeho Hwang and Uiwon Hwang},
  title = {{When Does Exploration Prevent Collapse? Three Regimes and the Path of the Reward in Self-Consistency Training}},
  year = {2026},
  eprint = {2611.01307},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01307},
}
Figure