Abstract
Many sequence-to-sequence dialogue models tend to generate safe, uninformative responses. Existing solutions often rely on static features or complex inference-time adjustments. We propose AvgOut, a dynamic metric that calculates the average output probability distribution across decoder time steps during training. This metric allows models to become aware of "dull" tokens—those with disproportionately high generation probabilities—without manual feature engineering. We introduce three training strategies leveraging AvgOut: MinAvgOut (regularization), Label Fine-Tuning (LFT), and Reinforcement Learning (RL). Our experiments on the Ubuntu Dialogue Corpus demonstrate that these models significantly improve both response diversity and technical relevance, outperforming standard LSTM baselines and competitive models like VHRED and MMI.
Introduction
A persistent challenge in neural dialogue systems is the tendency of sequence-to-sequence (seq2seq) models to produce "safe" but uninformative responses, such as "I don't know" or "I'm not sure." This phenomenon, known as low corpus-level diversity, occurs because models are optimized to maximize the likelihood of a response, leading them to gravitate toward statistically frequent but semantically empty phrases.
Previous mitigation strategies have typically involved hand-crafted lists of dull words or specialized decoding algorithms like Maximum Mutual Information (MMI). However, these approaches are often static and fail to account for how a model's internal biases evolve during training. We propose a dynamic, feature-free approach that monitors the model's own output distributions to identify and penalize the emergence of dullness in real-time.
The AvgOut Metric
The core of our approach is the AvgOut (Average Output Probability Distribution) metric. Unlike discrete metrics that only consider the final tokens chosen, AvgOut examines the entire probability distribution across the vocabulary at every decoding step.
We maintain an exponential moving average ($D$) of these distributions across training batches. If the model consistently assigns high probability to a specific set of tokens regardless of context, $D$ will exhibit sharp "peaks" at those tokens, identifying them as "dull."
Proposed Diversity-Promoting Models
We leverage the AvgOut signal through three distinct training paradigms:
- MinAvgOut (Regularization): This model directly maximizes diversity by penalizing similarity between the current batch distribution ($D'$) and the historical average ($D$). By minimizing the dot product $D \cdot D'$, the model is forced to distribute probability mass away from historically overused tokens.
- Label Fine-Tuning (LFT): We prepend a "diversity label" to the source sequence. During training, this label is continuously scaled based on the sequence's diversity score. This allows the model to learn a controllable mapping between the label and the resulting response diversity, enabling users to adjust the "diversity knob" during inference.
- Reinforcement Learning (RL): We treat the AvgOut score of a generated sequence as a reward signal. Using policy gradient methods, the model is encouraged to select more informative, rarer tokens that contribute to a higher diversity score.
Additionally, we evaluate a Hybrid model that combines the loss functions of MinAvgOut and RL to benefit from both global distribution regularization and sequence-level rewards.
Experimental Results
Models were evaluated on the Ubuntu Dialogue Corpus, a large-scale dataset of technical support interactions. Our models demonstrated a substantial performance leap over the standard LSTM baseline. For instance, the Hybrid model achieved an Activity F1 score of 8.05, a nearly seven-fold increase over the baseline's 1.18. This suggests that promoting diversity does not come at the cost of relevance; rather, it forces the model to utilize specific technical terms instead of generic filler.
Human evaluations further validated these findings. The LFT model was rated highest for "Plausibility" and "Content Richness," outperforming strong baselines like VHRED. As shown in the distribution analysis, our proposed models produce token usage patterns that more closely mirror the "long-tail" distribution of natural human speech compared to the highly peaked distributions of standard seq2seq models.
Conclusion
AvgOut provides a simple yet effective mechanism for building dynamically aware dialogue models. By treating dullness as a moving target in the probability space, we can effectively steer models toward more informative responses. While currently limited to unigram distributions, future work will explore scaling this approach to higher-order n-grams and testing its efficacy in non-technical, open-domain conversational settings.
Summary updated