Abstract
Standard neural dialog models often suffer from the "dull response" problem, generating generic and repetitive phrases like "I don't know." This paper proposes a boosting framework for dialog generation to improve response diversity and relevance. By iteratively training an ensemble of models that focus on data points previously handled poorly, the authors demonstrate significant improvements in conversation variety. Experimental results on the Persona dataset show that boosted models outperform baselines in both automated diversity metrics and human evaluation, without sacrificing coherence.
Introduction
Neural sequence-to-sequence models are the backbone of modern chatbots, yet they frequently produce uninspiring, generic responses. This occurs because standard training objectives maximize the likelihood of the next word, causing models to gravitate toward the most frequent patterns in the training data. While techniques like Maximum Mutual Information (MMI) and Reinforcement Learning have been explored to mitigate this, they often fail to capture the full breadth of human conversational modes, often missing the "long tail" of interesting responses.
This research investigates the application of boosting—a classic ensemble technique—to generative dialog models. The goal is to force models to attend to the "missing modes" of informative responses that are typically ignored by standard maximum likelihood estimation (MLE).
Methodology
The authors introduce a principled framework for boosting text generation models through the following components:
Iterative Training and Data Reweighting
Instead of a single model, a sequence of models is trained. Each subsequent model focuses on training examples that previous iterations failed to generate diversely. Since defining "error" in generation is non-trivial, the authors employ a rule-based discriminator. This discriminator tracks frequently generated generic responses; training examples with high n-gram overlap with these "dull" outputs are down-weighted in the next iteration. This mechanism prevents the ensemble from over-relying on safe, repetitive phrases.
Hybrid Objectives and Ensemble Decoding
The boosting framework is integrated with existing techniques to further enhance performance:
- Maximum Mutual Information (MMI): Used to penalize generic outputs during the decoding phase.
- Reward-Augmented Maximum Likelihood (RAML): Incorporates rewards (e.g., tf-idf matching) directly into the training objective to encourage the inclusion of content words.
At test time, candidates are generated by the strongest individual model and then re-ranked using z-normalized scores from the entire boosted ensemble.
Experimental Results
The approach was evaluated on the Persona Dataset, which provides specific context for conversations to ensure grounded responses.
Diversity and Relevance
The boosted models demonstrated substantial gains across multiple metrics. As shown in Figure 3, diversity metrics such as Inertia (a measure of embedding spread) and the count of distinct unigrams and trigrams increased significantly over five boosting iterations. Notably, the number of distinct trigrams nearly doubled, indicating a much wider vocabulary of phrases.
Furthermore, the relevance of generated responses improved. Figure 4 illustrates the steady increase in relevance metrics (such as cosine similarity to ground truth) as the boosting iterations progressed, particularly for the MLE and MMI-based variants.
Human Evaluation
Human judges consistently preferred the boosted models. In head-to-head comparisons, the boosted MLE model was preferred 37.6% of the time compared to 17.6% for the baseline. For RAML-based models, the preference for the boosted version rose to 44.8%, highlighting the synergy between boosting and reward-augmented training.
Discussion and Limitations
A key insight from this work is the failure of neural discriminators for data reweighting; simpler n-gram heuristics proved more robust against overfitting. Additionally, boosting works well with RAML by ensuring the model explores a variety of content words rather than fixating on a narrow set of keywords.
However, the approach has limitations:
- Computational Complexity: Training and performing inference with an ensemble is more resource-intensive than a single model.
- Heuristic Dependency: The rule-based discriminator relies on exact n-gram matches, which may miss semantic similarities.
- Scope: The current study is limited to single-turn dialogs; future work is needed to address multi-turn consistency.
Conclusion
Boosting provides a powerful mechanism for overcoming the dull response problem in dialog generation. By forcing models to explore the diverse modes of human conversation, the proposed framework significantly enhances the quality and variety of AI-generated responses.
Summary updated