TL;DR
Unsupervised Reinforcement Learning with Verifiable Rewards (RLVR) often fails because models learn to output the same answer for every task—a phenomenon called policy collapse. This paper reveals that collapse is a "collective instability" driven by shared parameters: as the number of training problems increases, a global bias grows faster than the model can sharpen its answers for individual tasks.
What Problem Does This Paper Address?
Training reasoning models (like those used for math or coding) typically requires "verifiable rewards," where a model is rewarded if its final answer matches a ground-truth label. However, labels are expensive. "Label-free" or unsupervised RLVR attempts to bypass this by rewarding the model for agreeing with its own majority vote across multiple samples.
While this works initially, it eventually leads to policy collapse: the model's self-assessed reward hits 1.0, but its actual accuracy plummets because it starts giving the same answer (e.g., always "+" or always "1") to every single problem. Curiously, empirical studies have shown that this collapse is more likely to happen when training on larger datasets. This paper provides the first theoretical explanation for why more data can actually destabilize self-reinforcement.
Methods
The authors approach the problem using tools from statistical physics and neural network theory:
- Minimal Many-Problem Model: They analyze a system of $N$ problems that share a single parameter (representing a global bias, like a preference for short answers).
- Linear Stability Analysis: They show that when a model is "undecided" (logits are near zero), any small shared perturbation is amplified. This is analogous to the paramagnetic instability in the Curie-Weiss model of magnetism.
- The "Race" Theory: They frame collapse as a race between two processes:
- Local Sharpening: The model getting more confident in its current (hopefully correct) answer for a specific task. This rate slows down as $N$ increases because each task is seen less frequently.
- Shared Mode Growth: A global bias that grows at a rate independent of $N$ because it aggregates signals from all tasks simultaneously.
- Kernel Formulation: They extend the theory to real neural networks using the Neural Tangent Kernel (NTK), identifying "protective coupling" (generalization) that can counteract collapse.
- Verification: The theory is tested via stochastic simulations and by training a small Transformer on a synthetic reasoning task.
Datasets
- Solvable Toy Model: A synthetic environment with $N$ binary classification problems used to precisely test the scaling laws of collapse.
- Six-Digit Sum Comparison: A reasoning task for the Transformer where the model must decide if the sum of the first three digits is greater than the sum of the last three. This provides a controlled setting to measure kernel drift and collapse in a real architecture.
Key Insights and Ideas
- The Dataset-Size Threshold: The most counterintuitive finding is that there is a critical number of problems, $N_c$, above which collapse is inevitable. Because local sharpening is diluted by $1/N$ while global bias is not, larger datasets give the "shared mode" more time to win the race and flip the model's answers.
- The Unstable Window: Models are only susceptible to collapse when their logits are within a specific range ($|z| < 1.32$ for $K=3$ samples). Once a model becomes sufficiently confident in its answers, it enters a stable regime where it can no longer be easily pushed into a global bias.
- Generalization as Protection: In the kernel view, "prediction-aligned coupling" (where learning one task helps solve another correctly) acts as a shield. If a network generalizes well, it is less likely to collapse.
- Entropy is a Symptom, Not the Cause: While high-entropy (less confident) models collapse faster, the authors argue that the true driver is the positive slope of the self-reinforcement drift, which amplifies shared noise.
Key Results
- Universal Scaling Law: The authors derive a closed-form expression for the critical dataset size: $N_c \simeq \ln(1/\delta) / (\lambda^2 G)$, where $\lambda$ is the coupling strength and $G$ relates to initial confidence.
- Inverse Square Scaling: Simulations confirm that $N_c \propto \lambda^{-2}$ (Figure 2c). A single critical amplification factor, $\Lambda_c \approx 2.6$, predicts collapse across various couplings and confidence profiles.
- Safety in Confidence: Confident initial models ($a > z^*$) were found to be stable and never collapsed, even with very large $N$ and long training times.
- Architectural Remedy: In Transformers, removing the "shared direction" (e.g., via an antisymmetric readout that cancels out global biases) prevented collapse entirely.
Limitations and Future Work
- Kernel Drift: The theory assumes a fixed kernel ("lazy training"), but real networks learn features. The authors observed that feature learning often erodes protective coupling, making models collapse more easily than the initial kernel would predict.
- Entropy Bonuses: While adding a marginal-entropy bonus can stop the model from outputting a single answer, it doesn't necessarily preserve accuracy; the model may simply collapse into a different, task-unrelated collective mode.
- Scale: The experiments were limited to small Transformers. Testing these diagnostics (like measuring the population-averaged slope $\langle F' \rangle$) on large-scale pretrained models is a crucial next step.
- Curriculum Learning: The theory suggests that training on confident problems first might prevent collapse, but this "curriculum by confidence" remains to be tested.
Summary updated