TL;DR
Alignment techniques often inadvertently train Large Language Models (LLMs) to give shorter, more homogeneous answers due to a hidden "brevity bias" in standard reward models and diversity metrics. To fix this, the authors introduce Diverse, not Short (Diverse-NS), a length-controlled data selection strategy that uses just 3,000 preference pairs to significantly boost response diversity without sacrificing quality or text length.
What Problem Does This Paper Address?
While alignment techniques like RLHF and Direct Preference Optimization (DPO) make LLMs safer and more helpful, they come with a well-documented side effect: they drastically reduce the diversity of model outputs. Aligned models tend to produce homogenized, less creative text, which can lead to model collapse over repeated generations.
The authors uncover a critical, overlooked reason for this: brevity bias. Common diversity metrics (like entropy) and reward models (like ArmoRM) are systematically, negatively correlated with text length.
When algorithms optimize for diversity without controlling for length, they simply teach the model to generate shorter text. This paper tackles the challenge of breaking this correlation to improve true lexical and semantic diversity while maintaining length parity.
Methods
The authors propose Diverse, not Short (Diverse-NS), a pipeline that curates high-quality preference pairs for DPO training using a two-step approach:
- Sequential Prompting: The model first generates a natural response ($r_1$). It is then prompted a second time to rewrite the story to be completely different in plot and characters, avoid standard openings, but match the exact word count of $r_1$, producing a more diverse candidate ($r_2$).
- Strict Data Filtering: To ensure the model learns positive traits, a preference pair is only kept if $r_2$ (chosen) has higher diversity and quality than $r_1$ (rejected), and their actual length difference is within $\pm 5$ words.
The standard Diverse-NS uses token entropy for diversity and ArmoRM scores for quality. Because running these models for filtering is computationally expensive (requiring massive forward passes), the authors also introduce Diverse-NS-Lite. This lightweight variant swaps out the heavy models for cheap CPU-computed proxies: Type-Token Ratio (TTR) for diversity and the MAAS index for quality.
Additionally, the authors introduce a new evaluation metric, $\Delta$ Diversity Decile ($\Delta$DD), which normalizes diversity scores against a lookup table of text lengths to ensure longer or shorter responses aren't unfairly penalized or credited.
Datasets
The authors curated and released a high-quality dataset of 6,000 preference pairs generated from LLaMA-3.1-8B and Olmo-2-7B.
To evaluate performance, they tested the models across four distinct creative generation benchmarks:
- Divergent Associations Task (DAT): Generating 10 completely dissimilar words.
- Persona Generation Task (PGT): Generating varied JSON objects containing names, cities, and occupations.
- Alternate Uses Task (AUT): Finding unconventional uses for everyday objects.
- Creative Writing Task (CWT): Writing short stories incorporating specific three-word sets.
Key Insights and Ideas
- The Brevity Bias Trap: Prior state-of-the-art diversity training methods like DivPO inadvertently shrink model outputs. By failing to control for length, DivPO selected responses that were, on average, 49.9 words shorter than the baseline. Diverse-NS keeps this delta close to 1 word.
- Smaller Models as "Diversity Teachers": In a surprising twist, the authors found that preference data generated by a smaller model (Olmo-2-7B) could successfully fine-tune a larger model (Olmo-2-13B), yielding significant diversity gains. This opens up low-cost paths for scaling diversity alignment.
- Proxies Work Remarkably Well: The Lite version of the pipeline completely eliminates the need for expensive reward model forward passes during filtering, yet it achieves virtually identical performance to the full pipeline.
Key Results
When applied to LLaMA-3.1-8B and the Olmo-2 family, Diverse-NS consistently outperformed both the base models and the DivPO baseline across all tasks.
Quantitatively, on the Creative Writing Task, Diverse-NS achieved statistically significant improvements in lexical diversity metrics (TTR, MTLD) and semantic distance (DSI). Human evaluators strongly preferred the outputs of Diverse-NS, giving it a win rate of up to 72.73% against DivPO.
Limitations and Future Work
While highly effective, the framework currently relies on single metrics for filtering during data curation, which might miss broader syntactic nuances. The authors also noted that despite strict filtering, the fine-tuned models naturally learned to become more expressive and slightly increased their average output length over time. Future work will focus on expanding this strategy to a multi-task setup that explicitly balances both short- and long-form generation tasks across multiple iterative rounds of self-training.
Summary updated