AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses

Tong Niu, Mohit Bansal

Abstract

Many sequence-to-sequence dialogue models tend to generate safe, uninformative responses. There have been various useful efforts on trying to eliminate them. However, these approaches either improve decoding algorithms during inference, rely on hand-crafted features, or employ complex models. In our work, we build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. Specifically, we start with a simple yet effective automatic metric, AvgOut, which calculates the average output probability distribution of all time steps on the decoder side during training. This metric directly estimates which tokens are more likely to be generated, thus making it a faithful evaluation of the model diversity (i.e., for diverse models, the token probabilities should be more evenly distributed rather than peaked at a few dull tokens). We then leverage this novel metric to propose three models that promote diversity without losing relevance. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch; the second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level; the third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal. Moreover, we experiment with a hybrid model by combining the loss terms of MinAvgOut and RL. All four models outperform their base LSTM-RNN model on both diversity and relevance by a large margin, and are comparable to or better than competitive baselines (also verified via human evaluation). Moreover, our approaches are orthogonal to the base model, making them applicable as an add-on to other emerging better dialogue models in the future.

Introduction

In conversational AI, models frequently fall into the "safe response" trap, producing generic utterances like "I don't know" or "That's interesting." While contextually plausible, these responses lack information and stall engagement. Standard sequence-to-sequence (seq2seq) training maximizes the likelihood of the next token, which inadvertently favors these high-frequency, low-risk phrases found in training data.

Previous solutions have relied on complex decoding strategies (e.g., Maximum Mutual Information), hand-crafted "dull word" lists, or intricate architectures like Variational Autoencoders (VAEs). However, static lists fail to adapt as model biases shift, and complex architectures can be difficult to train. This paper introduces AvgOut, a dynamic, model-aware metric that identifies dullness based on the model's own internal probability distributions.

The AvgOut Metric

The core innovation is the Average Output Probability Distribution (AvgOut). Unlike metrics that look only at the final token choice, AvgOut examines the entire probability distribution across the vocabulary at every decoding step.

Visualization of the AvgOut metric calculation, showing how the global average (D) and current batch (D') are combined.

By maintaining an exponential moving average ($D$) of these distributions across training batches, the system tracks which tokens the model is currently favoring. If a token's probability spikes in the global average, it is flagged as potentially "dull" or overused. This allows for a dynamic penalty that evolves alongside the model's learning process.

Diversity-Promoting Models

The authors propose three distinct ways to integrate AvgOut into the training process:

  1. MinAvgOut: A regularization approach that adds a loss term based on the dot product between the current batch's distribution ($D'$) and the global average ($D$). This directly penalizes the model for producing distributions that align too closely with its existing biases.
  2. Label Fine-Tuning (LFT): This method prepends a continuous "diversity label" to the source sequence. During training, this label is determined by the diversity of the ground-truth response. At inference, users can adjust this label to control the desired level of response diversity.
  3. Reinforcement Learning (RL): The AvgOut score is treated as a reward signal. Sequences that contain tokens with low probability in the global distribution $D$ receive higher rewards, encouraging the exploration of the "long tail" of the vocabulary.

A Hybrid model combining MinAvgOut and RL was also developed, aiming to leverage both stable distribution-level regularization and sequence-level reward optimization.

Experimental Results

The models were evaluated on the Ubuntu Dialogue Corpus, a large-scale dataset of technical support logs. Evaluation metrics included Activity and Entity F1 (for relevance) and Distinct-1/2 (for diversity).

Trigram diversity curves showing the flatter probability distribution achieved by the proposed models compared to the LSTM baseline.

The results demonstrate that pushing for diversity does not sacrifice relevance; in fact, it improves it. The Hybrid model achieved an Entity F1 of 5.47, significantly higher than the baseline LSTM (1.81) and the VHRED baseline (3.52). Human evaluations also favored the LFT model for its plausibility and content richness. As shown in the diversity curves, the proposed methods (especially the Hybrid and MinAvgOut variants) result in a much flatter distribution of trigrams, indicating a more varied and less repetitive vocabulary usage.

Conclusion

AvgOut provides a simple, effective, and model-agnostic way to improve dialogue diversity. By making the model aware of its own biases during training, researchers can eliminate dull responses without the need for manual feature engineering. These methods are orthogonal to the underlying architecture, making them easily applicable to future state-of-the-art dialogue models.

Summary updated

Abstract

Many sequence-to-sequence dialogue models tend to generate safe, uninformative responses. There have been various useful efforts on trying to eliminate them. However, these approaches either improve decoding algorithms during inference, rely on hand-crafted features, or employ complex models. In our work, we build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. Specifically, we start with a simple yet effective automatic metric, AvgOut, which calculates the average output probability distribution of all time steps on the decoder side during training. This metric directly estimates which tokens are more likely to be generated, thus making it a faithful evaluation of the model diversity (i.e., for diverse models, the token probabilities should be more evenly distributed rather than peaked at a few dull tokens). We then leverage this novel metric to propose three models that promote diversity without losing relevance. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch; the second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level; the third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal. Moreover, we experiment with a hybrid model by combining the loss terms of MinAvgOut and RL. All four models outperform their base LSTM-RNN model on both diversity and relevance by a large margin, and are comparable to or better than competitive baselines (also verified via human evaluation). Moreover, our approaches are orthogonal to the base model, making them applicable as an add-on to other emerging better dialogue models in the future.
BibTeX
@inproceedings{niu2020avgout,
  author = {Tong Niu and Mohit Bansal},
  title = {{AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=nexIq3Cp67U},
}
Figure