Exemplar Guided Neural Dialogue Generation

Hengyi Cai, Hongshen Chen, Yonghao Song, Xiaofang Zhao, Dawei Yin

Introduction: The Challenge of Generic Responses

A persistent issue in neural dialogue systems is the production of "safe responses"—generic, uninformative phrases like "I don't know" or "I'm not sure." While retrieval-augmented models attempt to solve this by leveraging existing training data, they frequently struggle with "noisy" exemplars that share surface-level similarities but lack topical relevance. For instance, a query about hobbies might retrieve a training sample about soccer when the user is actually discussing art, leading to a breakdown in conversational coherence.

Figure 1: Comparison between standard SEQ2SEQ and exemplar-guided generation. The model filters out topically irrelevant 'soccer' examples in favor of 'drawing' to produce a specific response about 'painting'.

The EnD Model Architecture

To address these limitations, the proposed EnD (Exemplar Guided Neural Dialogue Generation) framework introduces a sophisticated two-stage approach to response generation that prioritizes both textual and topical alignment.

Figure 2: The EnD framework architecture, illustrating the flow from input context (x) through the Exemplary Conversation Retriever to the Exemplar Guided Response Generation.

1. Two-Stage Exemplar Retrieval

The retrieval process ensures that the model draws from the most relevant "experiences" in the training set through a hierarchical filter:

  • Initial Filtering: BM25 is used for rapid keyword-based retrieval to identify a candidate subset of conversations based on literal word overlap.
  • Topical Refinement: A Neural Variational Topic Model then ranks these candidates by inferring latent topics. This step is crucial for distinguishing between contexts that look similar grammatically but differ semantically, ensuring the model doesn't talk about sports when the user is talking about art.

2. Exemplar-Guided Generation

The generation phase utilizes a multi-source sampling mechanism designed to move beyond simple sequence-to-sequence logic. The decoder integrates information from three distinct channels:

  • Standard Vocabulary: The model's base language knowledge.
  • Local Exemplar Guidance: Specific words from the retrieved exemplar are treated as a "soft template," providing structural and semantic cues.
  • Global Topic Guidance: Broader topic-related words ensure the response remains within the correct thematic domain.

A dynamic Gating Mechanism manages these channels at each time step. By calculating a probability distribution across these sources, the gate decides whether to "copy" a word from the exemplar or "generate" a word from the internal vocabulary. This flexibility allows the model to mimic the helpful structure of a past conversation while tailoring the specific details to the current user input.

Experimental Evaluation

The researchers evaluated EnD on a combined open-domain corpus consisting of movie-related discussions from Reddit and technical support logs from the Ubuntu Technical Corpus.

Key Results

  • Diversity and Informativeness: EnD significantly outperformed standard models in response variety. On the Distinct-3 metric (measuring unique 3-word phrases), EnD achieved a score of 10.33, compared to just 1.984 for the baseline SEQ2SEQ model.
  • Human Preference: In side-by-side human evaluations, EnD's responses were preferred over the baseline 57% of the time, with a loss rate of only 21%.
  • Ablation Studies: Results confirmed that both textual similarity and topical inference are essential; removing either component led to a measurable decline in performance, proving that literal similarity is a poor proxy for conversation flow on its own.

Conclusion

By treating training samples as "soft templates" and prioritizing topical relevance over simple word overlap, the EnD model effectively mitigates the safe response problem. This approach provides a robust framework for generating diverse, contextually grounded dialogue in open-domain systems, though future work may need to address the computational latency inherent in neural topic inference.

Summary updated

Abstract

Humans benefit from previous experiences when taking actions. Similarly, related examples from the training data also provide exemplary information for neural dialogue models when responding to a given input message. However, effectively fusing such exemplary information into dialogue generation is non-trivial: useful exemplars are required to be not only literally-similar, but also topic-related with the given context. Noisy exemplars impair the neural dialogue models understanding the conversation topics and even corrupt the response generation. To address the issues, we propose an exemplar guided neural dialogue generation model where exemplar responses are retrieved in terms of both the text similarity and the topic proximity through a two-stage exemplar retrieval model. In the first stage, a small subset of conversations is retrieved from a training set given a dialogue context. These candidate exemplars are then finely ranked regarding the topical proximity to choose the best-matched exemplar response. To further induce the neural dialogue generation model consulting the exemplar response and the conversation topics more faithfully, we introduce a multi-source sampling mechanism to provide the dialogue model with both local exemplary semantics and global topical guidance during decoding. Empirical evaluations on a large-scale conversation dataset show that the proposed approach significantly outperforms the state-of-the-art in terms of both the quantitative metrics and human evaluations.
BibTeX
@inproceedings{cai2020exemplar,
  author = {Hengyi Cai and Hongshen Chen and Yonghao Song and Xiaofang Zhao and Dawei Yin},
  title = {{Exemplar Guided Neural Dialogue Generation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/498},
}
Figure