Reinforcement learning (RL) has emerged as a powerful paradigm for upgrading generative models—turning Large Language Models (LLMs) from simple text predictors into capable reasoners, and diffusion models into high-fidelity artists. By rewarding models when they successfully solve a math problem, generate functional code, or align with human preferences, algorithms like PPO and GRPO dramatically boost single-shot accuracy.
However, this optimization process comes with a severe, often hidden cost: diversity collapse (also known as mode collapse). As a model learns to maximize rewards, its output distribution narrows dramatically. Instead of retaining multiple valid strategies to solve a problem or express a concept, the model converges on a single, repetitive sequence of tokens, actions, or styles. This loss of strategic variety creates a critical bottleneck for test-time scaling, multi-sample verification, and creative exploration.
The Mathematical Root: Why RL Seeks Modes
To understand why diversity collapse happens, we must look at the objective functions driving modern RL alignment. Most standard RL frameworks optimize for expected reward while using a Kullback-Leibler (KL) divergence penalty against a reference policy to prevent the model's parameters from drifting too far.
As explained by Li et al., standard policy optimization relies on reverse KL divergence. Reverse KL is inherently "mode-seeking." Mathematically, it heavily penalizes a model for placing probability mass in regions where the target reward is low (i.e., generating an incorrect or low-reward answer). However, it does not heavily penalize the model for completely ignoring other high-reward regions of the solution space.
Standard RL (Reverse KL): Mode-Seeking
Probability Mass
▲
│ █
│ █
│ █
───┴───────█───────────► Solution Space
(Drops alternative valid paths)
Distribution Matching (Forward KL): Mode-Covering
Probability Mass
▲
│ ▄ █ ▄
│ █ █ █
│ █ █ █
───┴───┴───┴───┴───────► Solution Space
(Preserves diverse solution paths)
Consequently, the policy learns to play it safe, concentrating all its probability mass on a narrow set of high-scoring, reliable paths (Kirk et al.). The model identifies a single high-reward trajectory and ignores other valid solutions (Yu et al.).
This phenomenon is not unique to text. In text-to-image generation and diffusion models, fine-tuning to maximize specific aesthetic or human preference rewards causes a similar "stylistic collapse" or reward over-optimization. The model drops the rich variety of its original data distribution to repeatedly exploit a single high-scoring visual pattern (Kim; Yun et al.).
The Test-Time Bottleneck: Why Variety Matters
Why is a lack of diversity problematic if the model is still getting the answer right? The issue becomes glaringly apparent when evaluating the model's capacity for complex problem-solving and test-time scaling.
Modern inference strategies often rely on generating multiple samples (e.g., pass@k or majority voting) to find the correct solution to difficult problems. However, Song et al. demonstrate that when a model suffers from diversity collapse during RL training, sampling multiple outputs during inference fails to yield distinct problem-solving strategies. Instead, standard stochastic sampling merely produces highly similar trajectories that differ only by minor token variations or synonymous phrasing, offering no real semantic diversity (Xing et al.).
Furthermore, this lack of exploration generalizes poorly. When a model loses the ability to explore alternative paths on simpler, solved problems during training, it fails to discover the novel reasoning strategies required to solve more complex, unseen tasks at the frontier of its capabilities (Song et al.). It also cripples multi-step self-correction; as Puerto et al. note, smaller language models frequently repeat the same logical errors across sequential attempts because they lack the latent diversity to pivot to an alternative perspective.
Countering Collapse: Strategies for Diverse Optimization
To unlock the full potential of test-time scaling without sacrificing accuracy, researchers are developing alternative optimization frameworks that explicitly preserve solution variety.
Distribution Matching over Reward Maximization
Instead of forcing a policy to maximize expected rewards (which drives it toward a single peak), new approaches aim to match the full reward distribution. Li et al. introduce Distribution-Matching Policy Optimization (DMPO), which approximates forward KL divergence at the group level to compel the model to cover all valid solution modes. Similarly, Xue proposes FlowRL, adapting principles from Generative Flow Networks (GFlowNets) to sample reasoning trajectories with a probability proportional to their rewards rather than greedily exploiting the single best path (Jain et al.).
Learning from Mistakes via Negative Reinforcement
Standard reinforcement learning with verifiable rewards often focuses heavily on reinforcing correct answers. However, Zhu et al. discovered that solely penalizing incorrect responses—termed Negative Sample Reinforcement—is remarkably effective at refining a model's logic. By focusing the training signal on what not to do, the algorithm leaves the model's creative and strategic diversity intact, avoiding the collapse typical of positive-only reinforcement.
Explicit Diversity Metrics and Parameter-Space Optimization
Other approaches tackle the problem by measuring and curating diversity directly. Ju et al. introduce Reasoning Path Divergence (RPD), a metric designed to quantify semantic differences between step-by-step reasoning chains. This allows practitioners to curate "one problem, multiple solutions" datasets that explicitly teach models to explore alternative paths.
Moving away from action-space optimization entirely, the authors of Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies show that optimizing in parameter space via Evolution Strategies broadens solution coverage and prevents the distribution collapse inherent to policy gradient methods. By shifting how models are penalized and sampled, these techniques ensure that AI systems remain both accurate and adaptable.