Why Do Neural Dialog Systems Generate Short and Meaningless Replies? a Comparison between Dialog and Translation

Bolin Wei, Shuai Lu, Lili Mou, Hao Zhou, Pascal Poupart, Ge Li, Zhi Jin

TL;DR

Neural dialog systems often produce short, uninformative replies. This paper hypothesizes this is due to the inherent ambiguity in dialog, where one utterance can have many plausible responses. By artificially introducing this ambiguity into machine translation tasks, the authors successfully replicate the "short and meaningless" reply phenomenon, providing evidence for their conjecture.

What Problem Does This Paper Address?

Open-domain dialog systems, powered by deep learning models like sequence-to-sequence (Seq2Seq) networks, have become increasingly popular. However, a persistent issue is that these systems tend to generate overly short and generic replies, often referred to as "universal replies" (e.g., "I don't know," "Me too"). While various methods have been proposed to mitigate this, a clear empirical explanation for why this happens, especially in comparison to machine translation systems (which generally perform better), has been lacking. The core question is: why do Seq2Seq models, when applied to dialog, produce less informative outputs than when applied to translation?

Methods

The authors propose a novel approach to empirically test their hypothesis: mimicking the "unaligned" property of dialog in a machine translation setting. They achieve this by shuffling the source and target pairs in a standard machine translation dataset. This artificial shuffling creates a scenario where a single source sentence is paired with multiple, potentially unrelated, target sentences from the original dataset. This effectively simulates the dialog scenario where a single utterance (source) can have multiple equally plausible replies (targets), thus creating a more ambiguous conditional distribution $p(t|s)$. They then train and evaluate Seq2Seq models on this "shuffled" translation data and compare its performance to both standard machine translation and standard dialog systems.

Datasets

  • Machine Translation: WMT-2017 English-to-German dataset (5.8M samples).
  • Dialog: Cornell Movie-Dialogs Corpus dataset (221k samples).

The authors also mention using toy datasets for generating specific scenarios, though details are not elaborated.

Key Insights and Ideas

  • The "Unaligned Problem" in Dialog: The central idea is that dialog is inherently more ambiguous than translation. A single user utterance can legitimately be followed by many different, equally valid responses, each with a different meaning. This contrasts with translation, where there's typically a more direct semantic correspondence between source and target sentences.
  • Mimicking Ambiguity: The clever trick of shuffling source-target pairs in translation data provides a controlled way to introduce this "unaligned" property, allowing for a direct comparison.
  • Shortness as a Symptom of Ambiguity: The paper suggests that when a model faces a highly ambiguous target distribution (many plausible options), it defaults to shorter, safer, and less informative outputs to minimize error.
  • Seq2Seq's Sensitivity to Data Distribution: The experiments demonstrate that Seq2Seq models are sensitive to the nature of the conditional distribution they are trained on, and the "unaligned" property significantly impacts their output quality.

Key Results

  • BLEU Scores: Table 1 shows that shuffling the translation dataset drastically reduces BLEU scores, mirroring the low scores observed in dialog systems. With 100% shuffling, BLEU-4 drops to near zero.
  • Length, Negative Log-Probability, and Entropy: Table 2 reveals that the dialog system generates significantly shorter replies (81.3% of reference length) with lower negative log-probability and entropy compared to machine translation. Crucially, when the translation dataset is shuffled (especially at higher percentages), the generated sentences exhibit similar characteristics: reduced length, lower negative log-probability, and lower entropy. For instance, 100% shuffled translation yields replies that are only 74.0% of reference length, with significantly lower entropy.
  • R2 Correlation: Table 3 shows that dialog systems have a lower R2 correlation between time steps and hidden states, suggesting a weaker ability to track sequential information compared to standard machine translation. Shuffling the translation data also reduces this correlation, further supporting the link between ambiguity and degraded performance.

Limitations and Future Work

  • Artificial Mimicry: The authors acknowledge that artificially shuffling translation data is not a perfect replica of the complex nuances of natural dialog. However, they argue it captures a "salient difference" (the unaligned property) that is crucial for understanding the problem.
  • Future Development: The findings suggest that future dialog systems might benefit from incorporating more information (e.g., dialog context, knowledge bases) to reduce the ambiguity of plausible replies, thereby guiding the model towards more informative outputs.
  • Model Suitability: The study implies that Seq2Seq models are better suited for applications with more aligned source-target relationships, like translation or summarization, rather than highly ambiguous tasks like open-domain dialog.
  • Unexplained Result: The paper notes an inconsistent result where 100% shuffling leads to a surprisingly high R2 correlation in the encoder and decoder, for which they currently lack a clear explanation.

Summary updated

Abstract

This paper addresses the question: In neural dialog systems, why do sequence-to-sequence (Seq2Seq) neural networks generate short and meaningless replies for open-domain response generation? We conjecture that in a dialog system, due to the randomness of spoken language, there may be multiple equally plausible replies for one utterance, causing the deficiency of a Seq2Seq model. To evaluate our conjecture, we propose a systematic way to mimic the dialog scenario in machine translation systems with both real datasets and toy datasets generated elaborately. Experimental results show that we manage to reproduce the phenomenon of generating short and meaningless sentences in the translation setting.
BibTeX
@inproceedings{wei2019why,
  author = {Bolin Wei and Shuai Lu and Lili Mou and Hao Zhou and Pascal Poupart and Ge Li and Zhi Jin},
  title = {{Why Do Neural Dialog Systems Generate Short and Meaningless Replies? a Comparison between Dialog and Translation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8682634},
}
Figure