Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Training

Wangchunshu Zhou, Qifei Li, Chenle Li

Abstract

In this paper, we propose Inverse Adversarial Training (IAT) algorithm for training neural dialogue systems to avoid generic responses and model dialogue history better. In contrast to standard adversarial training algorithms, IAT encourages the model to be sensitive to the perturbation in the dialogue history and therefore learning from perturbations. By giving higher rewards for responses whose output probability reduces more significantly when dialogue history is perturbed, the model is encouraged to generate more diverse and consistent responses. By penalizing the model when generating the same response given perturbed dialogue history, the model is forced to better capture dialogue history and generate more informative responses. Experimental results on two benchmark datasets show that our approach can better model dialogue history and generate more diverse and consistent responses. In addition, we point out a problem of the widely used maximum mutual information (MMI) based methods for improving the diversity of dialogue response generation models and demonstrate it empirically.

Introduction

Neural dialogue systems have made significant strides, yet they continue to struggle with two persistent issues: the "Dull Response Problem" and inadequate Dialogue History Modeling. Standard training using Maximum Likelihood Estimation (MLE) often leads models to favor statistically frequent but uninformative responses like "I don't know" or "I'm not sure." Furthermore, many models fail to utilize the provided conversation history effectively, often producing the same generic output regardless of the preceding context.

While techniques like Maximum Mutual Information (MMI) have been proposed to address diversity, they often introduce unintended shortcuts. Models trained with MMI may simply repeat keywords from the user's last utterance to artificially inflate relevance scores without truly understanding the dialogue flow. To overcome these limitations, we introduce Inverse Adversarial Training (IAT), a method that leverages input perturbations to force models to become more context-sensitive and informative.

Methods: Inverse Adversarial Training

In traditional adversarial training, models are trained to be robust—maintaining consistent output despite small input perturbations. IAT inverts this logic. We argue that a high-quality, specific response should be sensitive to its context. If the dialogue history changes, the probability of a specific, relevant response should decrease. Conversely, if a response's probability remains stable despite history changes, it is likely a generic "dull" response.

The IAT Workflow

The IAT process involves three main components:

  1. Perturbation: The dialogue history is intentionally modified through techniques such as shuffling sentence order, dropping words, or substituting nouns and verbs.
  2. Reward System: The model receives a high reward if the likelihood of its response drops significantly when the history is perturbed. This indicates the response was specifically tailored to the original context.
  3. Penalty System: The model is penalized if it generates the same response for both the original and perturbed histories, discouraging the use of context-agnostic, generic phrases.
The IAT mechanism: penalizing generic responses (left) and rewarding relevant ones (right).

IAT acts as a "continual training" step, typically applied after a model has achieved basic fluency through standard MLE. It can be implemented in a supervised manner using ground-truth responses or a self-supervised manner using model-generated outputs.

Experimental Setup

We evaluated IAT on two major benchmarks:

  • OpenSubtitles: A large-scale dataset of movie scripts, testing the model's performance in open-domain, varied conversations.
  • DailyDialog: A high-quality dataset of daily life conversations with an average of 7.9 turns per dialogue, specifically chosen to test long-term history tracking.

Results and Discussion

Our experiments demonstrate that IAT significantly improves both context awareness and response diversity. In tests involving perturbed history, IAT models showed much higher sensitivity than baselines, confirming they were actively processing the dialogue history.

Quantitative and Human Evaluation

On the DailyDialog dataset, IAT achieved a Distinct-3 score of 14.93, a substantial improvement over the vanilla Seq2Seq baseline (9.43). Beyond automated metrics, human evaluation by 20 graduate students confirmed the superiority of IAT across multiple dimensions.

Human evaluation scores for Fluency, Consistency, and Diversity across different models.

As shown in the human evaluation results, IAT outperformed all baselines, including MMI and CVAE variants, in Fluency (3.02), Consistency (3.05), and Diversity (3.34). Qualitatively, IAT models produced more engaging follow-ups. For instance, when a user mentioned a sore throat, the vanilla model offered a generic "I agree," while the IAT model asked a specific, relevant question about symptoms.

The MMI Shortcut

Our analysis also highlighted a flaw in MMI-based methods. We found that MMI models often exhibit high token overlap with the final utterance of the history. This suggests that instead of learning deep contextual relationships, these models learn to "parrot" the user to appear relevant, a shortcut that IAT successfully avoids by focusing on global history sensitivity.

Conclusion and Future Work

Inverse Adversarial Training provides a robust framework for reducing dull responses and improving context modeling in dialogue agents. By treating sensitivity as a proxy for quality, IAT forces models to move beyond statistical averages toward more meaningful interaction. Future work will focus on reducing the computational overhead of generating perturbations and addressing broader issues such as social bias and fairness in neural dialogue generation.

Summary updated

Abstract

In this paper, we propose Inverse Adversarial Training (IAT) algorithm for training neural dialogue systems to avoid generic responses and model dialogue history better. In contrast to standard adversarial training algorithms, IAT encourages the model to be sensitive to the perturbation in the dialogue history and therefore learning from perturbations. By giving higher rewards for responses whose output probability reduces more significantly when dialogue history is perturbed, the model is encouraged to generate more diverse and consistent responses. By penalizing the model when generating the same response given perturbed dialogue history, the model is forced to better capture dialogue history and generate more informative responses. Experimental results on two benchmark datasets show that our approach can better model dialogue history and generate more diverse and consistent responses. In addition, we point out a problem of the widely used maximum mutual information (MMI) based methods for improving the diversity of dialogue response generation models and demonstrate it empirically.
BibTeX
@inproceedings{zhou2021learning,
  author = {Wangchunshu Zhou and Qifei Li and Chenle Li},
  title = {{Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Training}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=uvEtrlfjk6z},
}
Figure