TL;DR
This paper introduces Inverse Adversarial Training (IAT), a novel method to train dialogue systems that are more informative and less generic. By training models to be sensitive to perturbations in dialogue history, IAT encourages them to better understand context and generate more diverse and consistent responses.
What Problem Does This Paper Address?
Modern neural dialogue systems, often trained with Maximum Likelihood Estimation (MLE), tend to produce generic and unengaging responses like "I don't know." This is partly due to the high frequency of such responses in training data and the over-simplified training objective. Furthermore, research indicates that these systems often fail to effectively utilize dialogue history, leading to irrelevant or inconsistent responses. This paper aims to tackle both the "dull response problem" and the poor dialogue history modeling in neural dialogue generation.
Methods
The core innovation is Inverse Adversarial Training (IAT). Unlike traditional adversarial training that aims to make models robust to adversarial attacks by generating the same output, IAT perturbs the dialogue history and trains the model to change its output. This encourages the model to be sensitive to the input context.
Specifically, IAT works by:
- Perturbing Dialogue History: The dialogue history (input context) is modified using various techniques at both the utterance level (shuffling, reversing, dropping, replacing utterances) and word level (shuffling, reversing, dropping, replacing words).
- Reward and Penalty Mechanism:
- Reward: The model is rewarded when it generates a response whose likelihood decreases significantly when the dialogue history is perturbed. This incentivizes generating informative and relevant responses.
- Penalty: The model is penalized when it generates a response whose likelihood barely changes given a perturbed history. This forces the model to better capture the nuances of the dialogue history.
- Sequence-Level Objective: The training objective is formulated at the sequence level, considering the variation in likelihood of generated responses given original versus perturbed dialogue history. This is framed as a reward function for dialogue generation.
- Supervised and Self-Supervised Variants: IAT can be applied in both supervised (using ground-truth responses) and self-supervised (generating responses) settings, making it flexible.
Datasets
The authors evaluate their approach on two datasets:
- OpenSubtitles: A large, open-domain dataset derived from movie scripts. Each turn is treated as a response, with the two preceding sentences as history.
- DailyDialog: A smaller, manually curated dataset of daily conversations across various topics. This dataset is noted for its longer average dialogue length, making it suitable for evaluating long-term history modeling.
Key Insights and Ideas
- Perturbation as a Teacher: The core idea that changing the input (dialogue history) should lead to a different output is a clever inversion of standard adversarial robustness training. It forces the model to learn what truly matters in the context.
- IAT Addresses Two Problems Simultaneously: The method is designed to combat both generic responses and poor history modeling, suggesting these issues are intertwined.
- MMI's "Shortcut" Problem: The paper highlights a limitation of Maximum Mutual Information (MMI) methods: they can lead to responses that repeat tokens from the last utterance to artificially boost scores, rather than generating truly informative content. This is empirically demonstrated.
- Dialogue History Matters: The research strongly suggests that current neural dialogue systems are often insensitive to dialogue history, a problem IAT aims to rectify.
Key Results
- Improved Dialogue History Modeling: IAT significantly outperforms baseline models in dialogue history modeling, as measured by the increase in perplexity when models are fed perturbed dialogue history. The IAT model shows the largest increase in perplexity across both datasets.
- Increased Diversity and Informativeness: IAT leads to more diverse responses, as indicated by higher distinct n-gram counts. Human evaluations also show IAT improves consistency and diversity, with fluency remaining comparable to baselines.
- Reduced Generic Responses: The method effectively reduces generic responses, as confirmed by human evaluations and metrics like stop-word percentage.
- Robustness: IAT does not negatively impact the model's adversarial robustness.
Limitations and Future Work
- Human Evaluation is Crucial: The authors acknowledge that automated metrics can be poor for evaluating dialogue systems and rely on human evaluation for key findings, which is resource-intensive.
- Ablation Study Insights: An ablation study revealed that both supervised and self-supervised components of IAT contribute to consistency and diversity. The reward mechanism (Eq. 3) is more impactful for diversity, while the penalty mechanism (Eq. 5) aids in history modeling and consistency. Both utterance-level and word-level perturbations are beneficial, with utterance-level perturbations showing a slight edge for consistency.
- Future Work: The paper suggests that IAT is orthogonal to other methods like re-ranking and can be combined with them. Further exploration of different perturbation strategies and their impact on specific dialogue phenomena could be valuable. The authors also briefly touch upon the broader societal impact of dialogue systems and the importance of controlling bias.
Summary updated