Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models

Vijeta Deshpande, Debasmita Ghose, John D. Patterson, Roger Beaty, Anna Rumshisky

TL;DR

Alignment techniques often inadvertently train Large Language Models (LLMs) to give shorter, more homogeneous answers due to a hidden "brevity bias" in standard reward models and diversity metrics. To fix this, the authors introduce Diverse, not Short (Diverse-NS), a length-controlled data selection strategy that uses just 3,000 preference pairs to significantly boost response diversity without sacrificing quality or text length.

What Problem Does This Paper Address?

While alignment techniques like RLHF and Direct Preference Optimization (DPO) make LLMs safer and more helpful, they come with a well-documented side effect: they drastically reduce the diversity of model outputs. Aligned models tend to produce homogenized, less creative text, which can lead to model collapse over repeated generations.

The authors uncover a critical, overlooked reason for this: brevity bias. Common diversity metrics (like entropy) and reward models (like ArmoRM) are systematically, negatively correlated with text length.

Table 1: Correlation Analysis. Pearson correlation coefficients between six text statistics and two target metrics: entropy (diversity) and ArmoRM reward scores (quality). Both entropy and ArmoRM scores show negative correlation with text length. Among diversity metrics, TTR exhibits the strongest correlation with entropy, while the MAAS index shows the highest correlation with ArmoRM scores. ∗∗: p < 0.001; all others: p < 0.0001.

When algorithms optimize for diversity without controlling for length, they simply teach the model to generate shorter text. This paper tackles the challenge of breaking this correlation to improve true lexical and semantic diversity while maintaining length parity.

Methods

The authors propose Diverse, not Short (Diverse-NS), a pipeline that curates high-quality preference pairs for DPO training using a two-step approach:

  1. Sequential Prompting: The model first generates a natural response ($r_1$). It is then prompted a second time to rewrite the story to be completely different in plot and characters, avoid standard openings, but match the exact word count of $r_1$, producing a more diverse candidate ($r_2$).
  2. Strict Data Filtering: To ensure the model learns positive traits, a preference pair is only kept if $r_2$ (chosen) has higher diversity and quality than $r_1$ (rejected), and their actual length difference is within $\pm 5$ words.

The standard Diverse-NS uses token entropy for diversity and ArmoRM scores for quality. Because running these models for filtering is computationally expensive (requiring massive forward passes), the authors also introduce Diverse-NS-Lite. This lightweight variant swaps out the heavy models for cheap CPU-computed proxies: Type-Token Ratio (TTR) for diversity and the MAAS index for quality.

Additionally, the authors introduce a new evaluation metric, $\Delta$ Diversity Decile ($\Delta$DD), which normalizes diversity scores against a lookup table of text lengths to ensure longer or shorter responses aren't unfairly penalized or credited.

Datasets

The authors curated and released a high-quality dataset of 6,000 preference pairs generated from LLaMA-3.1-8B and Olmo-2-7B.

To evaluate performance, they tested the models across four distinct creative generation benchmarks:

  • Divergent Associations Task (DAT): Generating 10 completely dissimilar words.
  • Persona Generation Task (PGT): Generating varied JSON objects containing names, cities, and occupations.
  • Alternate Uses Task (AUT): Finding unconventional uses for everyday objects.
  • Creative Writing Task (CWT): Writing short stories incorporating specific three-word sets.

Key Insights and Ideas

  • The Brevity Bias Trap: Prior state-of-the-art diversity training methods like DivPO inadvertently shrink model outputs. By failing to control for length, DivPO selected responses that were, on average, 49.9 words shorter than the baseline. Diverse-NS keeps this delta close to 1 word.
  • Smaller Models as "Diversity Teachers": In a surprising twist, the authors found that preference data generated by a smaller model (Olmo-2-7B) could successfully fine-tune a larger model (Olmo-2-13B), yielding significant diversity gains. This opens up low-cost paths for scaling diversity alignment.
  • Proxies Work Remarkably Well: The Lite version of the pipeline completely eliminates the need for expensive reward model forward passes during filtering, yet it achieves virtually identical performance to the full pipeline.

Key Results

When applied to LLaMA-3.1-8B and the Olmo-2 family, Diverse-NS consistently outperformed both the base models and the DivPO baseline across all tasks.

Figure 1: Diversity and Quality Evaluation on CWT. This figure shows ∆Diversity Decile (∆DD) values (y-axis) across various metrics (x-axis), computed from 70 CWT responses generated by the Olmo-2-7B model. A value of zero represents base model performance; bars indicate improvements from preference-tuned models. D-NS achieves the highest diversity gains overall, while D-NS-Lite consistently outperforms DivPO, except under TTR. In terms of quality (ArmoRM), DivPO shows a slight improvement, whereas our methods show a minor drop.

Quantitatively, on the Creative Writing Task, Diverse-NS achieved statistically significant improvements in lexical diversity metrics (TTR, MTLD) and semantic distance (DSI). Human evaluators strongly preferred the outputs of Diverse-NS, giving it a win rate of up to 72.73% against DivPO.

Limitations and Future Work

While highly effective, the framework currently relies on single metrics for filtering during data curation, which might miss broader syntactic nuances. The authors also noted that despite strict filtering, the fine-tuned models naturally learned to become more expressive and slightly increased their average output length over time. Future work will focus on expanding this strategy to a multi-task setup that explicitly balances both short- and long-form generation tasks across multiple iterative rounds of self-training.

Summary updated

Abstract

Diverse language model responses are crucial for creative generation, open-ended tasks, and self-improvement training. We show that common diversity metrics, and even reward models used for preference optimization, systematically bias models toward shorter outputs, limiting expressiveness. To address this, we introduce Diverse, not Short (Diverse-NS), a length-controlled data selection strategy that improves response diversity while maintaining length parity. By generating and filtering preference data that balances diversity, quality, and length, Diverse-NS enables effective training using only 3,000 preference pairs. Applied to LLaMA-3.1-8B and the Olmo-2 family, Diverse-NS substantially enhances lexical and semantic diversity. We show consistent improvement in diversity with minor reduction or gains in response quality on four creative generation tasks: Divergent Associations, Persona Generation, Alternate Uses, and Creative Writing. Surprisingly, experiments with the Olmo-2 model family (7B, and 13B) show that smaller models like Olmo-2-7B can serve as effective "diversity teachers" for larger models. By explicitly addressing length bias, our method efficiently pushes models toward more diverse and expressive outputs.
BibTeX
@inproceedings{deshpande2025diverse,
  author = {Vijeta Deshpande and Debasmita Ghose and John D. Patterson and Roger Beaty and Anna Rumshisky},
  title = {{Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure