Generating Dialogue Responses from a Semantic Latent Space

Wei-Jen Ko, Avik Ray, Yilin Shen, Hongxia Jin

Wei-Jen Ko, Avik Ray, Yilin Shen, Hongxia Jin. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

Introduction

Open-domain dialogue generation is a challenging "one-to-many" task where a single prompt can elicit numerous valid responses. Traditional models trained with Maximum Likelihood Estimation (MLE) and cross-entropy loss often struggle with this diversity. Because probability mass is distributed across many specific responses, models tend to favor high-frequency, "safe" generic responses like "I don't know" or "That's cool." This paper proposes a framework that shifts dialogue generation from word-level classification to a regression task within a shared semantic latent space.

Methodology: Regression in Latent Space

The core innovation is treating response generation as a regression problem. By aligning prompt and response representations, the model aggregates information from semantically similar responses, mitigating the generic response problem.

Learning the Latent Space

The authors utilize a modified Deep Canonical Correlation Analysis (DCCA) to create a shared space. Here, the representation of a prompt ($X$) and its correlated response ($Y$) are constrained to be close. This allows the model to recognize that semantically related but lexically different sentences (e.g., "I play the piano" and "I'm learning the guitar") belong to the same prompt context.

Correlated vs. Uncorrelated Information

The response representation is decomposed into two distinct components:

  1. Correlated ($Y$): Information directly tied to the prompt.
  2. Uncorrelated ($Y_u$): Information specific to the response but independent of the prompt (e.g., introducing a new topic). This is modeled using a Variational Autoencoder (VAE) with KL-regularization.
The model architecture during training and testing phases, highlighting the separation of correlated and uncorrelated information.

Decoding and Denoising

An autoencoder reconstructs the full response from these latent vectors. To ensure robustness against the noise inherent in continuous latent spaces, the authors employ a denoising technique, randomly replacing input words with an <unk> token during training. During inference, the model predicts a latent vector from the prompt and decodes it into a sentence.

Experimental Results

The model was evaluated on two major benchmarks: PersonaChat (persona-based chit-chat) and DailyDialog (everyday human conversations). The researchers ensured no overlap between training and test sets to rigorously test generalization.

Automatic metrics and human evaluation results on PersonaChat and DailyDialog datasets.

Key Findings

  • Human Evaluation: The model significantly outperformed baselines in "Useful Information" (UI) scores, which combine relevance and informativeness. On PersonaChat, it achieved a UI score of 1.97 compared to the MLE baseline's 1.43.
  • Diversity: Higher Dist-2 scores (distinct bigrams) indicate that the model generates more diverse and less repetitive language than traditional methods.
  • The "Generic Trap": t-SNE visualizations showed that generic responses sit far from specific responses in the latent space. By regressing toward clusters of specific responses, the model avoids falling into the "generic trap."

Discussion and Limitations

While the model excels in informativeness, there is a trade-off in grammaticality. Human evaluators rated approximately 71% of the generated sentences as perfectly grammatical, whereas MLE models, which produce simpler and safer sentences, maintain higher grammar scores. Additionally, the model exhibits high perplexity; however, the authors argue that perplexity is a poor metric for open-ended dialogue as it penalizes valid deviations from a single reference.

Future work may involve using larger pre-trained language models as decoders to improve grammaticality while retaining the advantages of the semantic latent space.

Summary updated

BibTeX
@inproceedings{ko2020generating,
  author = {Wei-Jen Ko and Avik Ray and Yilin Shen and Hongxia Jin},
  title = {{Generating Dialogue Responses from a Semantic Latent Space}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=SDk-4j15Zqy},
}
Figure