Jointly Reinforcing Diversity and Quality in Language Model Generations

Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, Tianlu Wang

Introduction: The Challenge of Diversity Collapse

Standard post-training techniques for Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF) or Group Relative Policy Optimization (GRPO), are designed to maximize accuracy and helpfulness. However, this optimization often leads to distribution sharpening—a phenomenon where the model converges on a narrow set of "safe," high-reward responses. This lack of diversity is a significant bottleneck for tasks requiring creativity or exploration, such as storytelling and complex problem-solving. Previous attempts to address this often relied on surface-level lexical diversity, which models can easily "hack" without introducing truly new ideas.

The DARLING Framework

To overcome these limitations, we introduce DARLING (Diversity-Aware Reinforcement Learning), a framework that integrates a semantic diversity signal directly into the online reinforcement learning loop.

The DARLING framework: from rollout generation to semantic clustering and multiplicative reward updates.

The DARLING process consists of three key steps:

  1. Semantic Partitioning: Instead of measuring simple word overlaps, DARLING utilizes a learned binary classifier to determine if two responses are "semantically equivalent." During training, rollouts for a single prompt are partitioned into clusters based on this semantic understanding.
  2. Diversity Reward: A diversity score is calculated for each response, reflecting how many other responses in the batch belong to different semantic clusters.
  3. Multiplicative Fusion: The final reward is defined as the product of the quality reward and the normalized diversity score: $r_{darling} = r_{quality} \times \text{Norm}(\text{Diversity})$. This multiplicative approach ensures that the model is only heavily rewarded for trajectories that are both high-quality and unique.

Evaluation and Datasets

DARLING was evaluated across two distinct domains:

  • Creative and Instruction Following Tasks: Using a 10k subset of WildChat, models were assessed on AlpacaEval 2.0, ArenaHard v2.0, EQ-Bench (for creative writing), and NoveltyBench (for semantic diversity).
  • Verifiable Math Tasks: Using 10k examples from DeepscaleR, models were tested on high-level competition benchmarks including AIME25, OlympiadBench, HMMT 2025, and Brumo 2025.

Key Insights: Diversity as a Catalyst for Quality

A core finding of this work is that explicitly rewarding diversity serves as an exploration catalyst, actually improving overall quality. In mathematical reasoning, diversity pressure forces the model to explore a wider range of paths, increasing the likelihood of discovering the correct solution.

Furthermore, the study highlights the failure of lexical metrics; rewarding n-gram diversity often leads to "reward hacking" (e.g., adding irrelevant punctuation). In contrast, semantic classifiers are much more robust. The choice of multiplicative fusion over additive reward also proved critical for balancing the two signals effectively.

Experimental Results

Experiments were conducted using Llama-3.1-8B, Llama-3.3-70B, and Qwen3 models.

Pareto front comparison of Quality vs. Diversity across different temperature settings.
  • Creative Writing: On EQ-Bench, DARLING (8B) achieved an ELO of 905, significantly outperforming the standard GRPO baseline of 659. It consistently produced more "interesting and original" content while avoiding clichés.
  • Instruction Following: On AlpacaEval 2.0, DARLING improved the length-controlled win rate (LCWR) from 48.7% to 55.2%.
  • Competition Math: DARLING improved pass@1 by an average of 3.51% for 4B models and 1.9% for 14B models, while significantly boosting solution variety (pass@128) by up to 10.16%.

As shown in the Pareto analysis, DARLING consistently pushes the performance frontier, achieving higher quality at every level of diversity compared to existing baselines.

Limitations and Future Directions

While effective, DARLING's performance is tied to the quality of its semantic classifier. Training such classifiers for specialized domains (e.g., legal or medical) remains a challenge. Additionally, the pairwise classification adds some computational overhead during training. Future work will explore extending DARLING to multi-turn conversations and other modalities like image generation, aiming to discover entirely new reasoning strategies in scientific domains.

Summary updated

BibTeX
@inproceedings{li2026jointly,
  author = {Tianjian Li and Yiming Zhang and Ping Yu and Swarnadeep Saha and Daniel Khashabi and Jason Weston and Jack Lanchantin and Tianlu Wang},
  title = {{Jointly Reinforcing Diversity and Quality in Language Model Generations}},
  booktitle = {SPOT @ ICLR},
  year = {2026},
  url = {https://openreview.net/forum?id=zW70rGMPuy},
}
Figure