AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses

Tong Niu, Mohit Bansal

Abstract

Many sequence-to-sequence dialogue models tend to generate safe, uninformative responses. Existing solutions often rely on static features or complex inference-time adjustments. We propose AvgOut, a dynamic metric that calculates the average output probability distribution across decoder time steps during training. This metric allows models to become aware of "dull" tokens—those with disproportionately high generation probabilities—without manual feature engineering. We introduce three training strategies leveraging AvgOut: MinAvgOut (regularization), Label Fine-Tuning (LFT), and Reinforcement Learning (RL). Our experiments on the Ubuntu Dialogue Corpus demonstrate that these models significantly improve both response diversity and technical relevance, outperforming standard LSTM baselines and competitive models like VHRED and MMI.

Introduction

A persistent challenge in neural dialogue systems is the tendency of sequence-to-sequence (seq2seq) models to produce "safe" but uninformative responses, such as "I don't know" or "I'm not sure." This phenomenon, known as low corpus-level diversity, occurs because models are optimized to maximize the likelihood of a response, leading them to gravitate toward statistically frequent but semantically empty phrases.

Previous mitigation strategies have typically involved hand-crafted lists of dull words or specialized decoding algorithms like Maximum Mutual Information (MMI). However, these approaches are often static and fail to account for how a model's internal biases evolve during training. We propose a dynamic, feature-free approach that monitors the model's own output distributions to identify and penalize the emergence of dullness in real-time.

The AvgOut Metric

The core of our approach is the AvgOut (Average Output Probability Distribution) metric. Unlike discrete metrics that only consider the final tokens chosen, AvgOut examines the entire probability distribution across the vocabulary at every decoding step.

We maintain an exponential moving average ($D$) of these distributions across training batches. If the model consistently assigns high probability to a specific set of tokens regardless of context, $D$ will exhibit sharp "peaks" at those tokens, identifying them as "dull."

Visualization of the AVGOUT metric comparing historical averages (D) with current batch distributions (D') to identify dull tokens.

Proposed Diversity-Promoting Models

We leverage the AvgOut signal through three distinct training paradigms:

  1. MinAvgOut (Regularization): This model directly maximizes diversity by penalizing similarity between the current batch distribution ($D'$) and the historical average ($D$). By minimizing the dot product $D \cdot D'$, the model is forced to distribute probability mass away from historically overused tokens.
  2. Label Fine-Tuning (LFT): We prepend a "diversity label" to the source sequence. During training, this label is continuously scaled based on the sequence's diversity score. This allows the model to learn a controllable mapping between the label and the resulting response diversity, enabling users to adjust the "diversity knob" during inference.
  3. Reinforcement Learning (RL): We treat the AvgOut score of a generated sequence as a reward signal. Using policy gradient methods, the model is encouraged to select more informative, rarer tokens that contribute to a higher diversity score.

Additionally, we evaluate a Hybrid model that combines the loss functions of MinAvgOut and RL to benefit from both global distribution regularization and sequence-level rewards.

Experimental Results

Models were evaluated on the Ubuntu Dialogue Corpus, a large-scale dataset of technical support interactions. Our models demonstrated a substantial performance leap over the standard LSTM baseline. For instance, the Hybrid model achieved an Activity F1 score of 8.05, a nearly seven-fold increase over the baseline's 1.18. This suggests that promoting diversity does not come at the cost of relevance; rather, it forces the model to utilize specific technical terms instead of generic filler.

Comparison of token distribution curves across different models; proposed methods track closer to the ground truth distribution.

Human evaluations further validated these findings. The LFT model was rated highest for "Plausibility" and "Content Richness," outperforming strong baselines like VHRED. As shown in the distribution analysis, our proposed models produce token usage patterns that more closely mirror the "long-tail" distribution of natural human speech compared to the highly peaked distributions of standard seq2seq models.

Conclusion

AvgOut provides a simple yet effective mechanism for building dynamically aware dialogue models. By treating dullness as a moving target in the probability space, we can effectively steer models toward more informative responses. While currently limited to unigram distributions, future work will explore scaling this approach to higher-order n-grams and testing its efficacy in non-technical, open-domain conversational settings.

Summary updated

Abstract

Many sequence-to-sequence dialogue models tend to generate safe, uninformative responses. There have been various useful efforts on trying to eliminate them. However, these approaches either improve decoding algorithms during inference, rely on hand-crafted features, or employ complex models. In our work, we build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. Specifically, we start with a simple yet effective automatic metric, AvgOut, which calculates the average output probability distribution of all time steps on the decoder side during training. This metric directly estimates which tokens are more likely to be generated, thus making it a faithful evaluation of the model diversity (i.e., for diverse models, the token probabilities should be more evenly distributed rather than peaked at a few dull tokens). We then leverage this novel metric to propose three models that promote diversity without losing relevance. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch; the second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level; the third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal. Moreover, we experiment with a hybrid model by combining the loss terms of MinAvgOut and RL. All four models outperform their base LSTM-RNN model on both diversity and relevance by a large margin, and are comparable to or better than competitive baselines (also verified via human evaluation). Moreover, our approaches are orthogonal to the base model, making them applicable as an add-on to other emerging better dialogue models in the future.
BibTeX
@inproceedings{niu2020avgout,
  author = {Tong Niu and Mohit Bansal},
  title = {{AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=nexIq3Cp67U},
}

Loading related content...

Figure