Diversity Promoting Objectives for Neural Text Generation

This direction is merged into another direction: repetition penalty. Its last version stays readable and citable here.

Version 1 of 136 papers47 conceptsfrozen 2026-10-04

Standard language models are often victims of their own training. When optimized solely via Maximum Likelihood Estimation (MLE) or word-overlap metrics like BLEU, models frequently fall into "repetition traps"—generating redundant, circular, or generic text. This happens because the model learns that repeating high-probability tokens is a "safe" way to minimize loss, even if it results in "word stuttering" or a "copy-paste" style that lacks human-like expressiveness.

To break these loops, researchers have moved beyond simple likelihood, developing objectives that explicitly penalize redundancy and reward novelty.

Breaking the Repetition Trap with Reinforcement Learning

Reinforcement Learning (RL) provides a flexible framework to define not just what a model should say, but what it should avoid. By treating repetition as a negative reward, we can force models to explore more diverse linguistic paths.

Luke Melas-Kyriazi and Alexander M Rush demonstrated that combining Self-Critical Sequence Training (SCST) with a simple trigram repetition penalty can nearly double performance scores in image paragraph captioning. The intuition is synergistic: the penalty blocks the "easy" path of repetition, forcing the RL agent to explore more descriptive content to find rewards.

This approach is particularly effective in specialized domains:

  • Visual Storytelling: Yu Cheng and Junjie Hu 0001 introduced ReCo-RL, which uses a composite reward function. It specifically penalizes n-gram overlaps between the current sentence and previous segments, ensuring the narrative remains "expressive" rather than repetitive.
  • Medical Reporting: In high-stakes tasks like X-ray interpretation, Chang Qi used repetition penalties to ensure reports are readable and free of redundant findings, preventing the model from "gaming" semantic similarity scores by repeating high-probability medical terms.
  • Reasoning Chains: Even in complex logic tasks, researchers like those behind Demystifying Long Chain-of-Thought Reasoning in LLMs found that repetition penalties are essential to prevent models from artificially inflating their reasoning length through token stuttering.

Unlikelihood Training and Negative Constraints

While RL modifies the reward landscape, unlikelihood loss modifies the training objective itself. Instead of only teaching the model what to predict, unlikelihood training explicitly teaches it what not to predict.

Jonas Kulhanek and Ondřej Dušek utilized this in the AuGPT framework to mitigate "hallucinations" and vocabulary collapse. By penalizing the model for assigning high probability to tokens that have recently appeared in the context, the model learns to distribute its probability mass more broadly. Similarly, Zeyang Lei applied unlikelihood loss to advertisement generation, successfully reducing the generic, repetitive "boilerplate" text that often plagues low-resource generative tasks.

In the context of non-autoregressive models—which generate all tokens in parallel and are prone to "word salad"—Yixuan Su used context-aware objectives to penalize the recurrence of the same token within a local window. This constraint acts as a grammatical stabilizer, ensuring that speed does not come at the cost of coherence.

Incentivizing Novelty and Abstraction

A common failure mode in summarization is the "copy-paste" problem, where models extract verbatim strings from the source rather than paraphrasing. Wojciech Kryscinski addressed this by introducing a novelty reward. This reward is calculated as the proportion of generated n-grams that do not appear in the source text. By explicitly rewarding the model for using its own "vocabulary," the system produces significantly more human-like, abstractive summaries.

This concept of "reading between the lines" is also explored by Zihao Fu, who implemented a penalty for repeating non-overlapping substrings. This prevents redundancy when a model merges external knowledge with primary input data, ensuring the output remains information-rich without being repetitive.

Dynamic Suppression during Decoding

If a model hasn't been trained with diversity in mind, researchers often apply "intervention" during the inference phase.

  • Logit Suppression: Ece Kamar and colleagues showed that penalizing tokens that have appeared frequently in previous outputs forces the model toward rarer vocabulary. While this can sometimes hurt accuracy, it is vital for generating diverse synthetic datasets.
  • Frequency Penalties: Seonil Son (GLAC Net) and Yao Dou (Scarecrow) utilize frequency-based penalties during word sampling. By evaluating a pool of candidates and discouraging recently used terms, these models produce narratives that feel less like a list of observations and more like a varied, human-written story.

Why These Penalties Work

The core intuition across these papers is that diversity is a constraint that enables better optimization. Without these penalties, models gravitate toward a "mode collapse" where a few high-frequency phrases dominate the output. By introducing a cost for repetition—whether through RL rewards (Guanlin Li), unlikelihood loss (Zeyang Lei), or sampling penalties (Yao Dou)—we force the model to explore the full breadth of its learned language distribution. The result is text that is not just more accurate, but more informative, engaging, and human-like.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_diversity_promoting_objectives_for_neura_18e1020c,
  title = {Diversity Promoting Objectives for Neural Text Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/diversity-promoting-objectives-for-neural-text-generation/rdr_18e1020cfcb149e0a09836dd3a2228aa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_diversity_promoting_objectives_for_neura_18e1020c_v1,
  title = {Diversity Promoting Objectives for Neural Text Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/diversity-promoting-objectives-for-neural-text-generation/rdr_18e1020cfcb149e0a09836dd3a2228aa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}