Improving Knowledge-Aware Dialogue Response Generation by Using Human-Written Prototype Dialogues

Sixing Wu, Ying Li, Dawei Zhang, Zhonghai WuEMNLP 2020 · OpenReview

Abstract

Open-domain generative dialogue systems often struggle with the "generic response" problem, frequently producing safe but uninformative replies like "I don't know." While grounding models in external knowledge graphs (KGs) can mitigate this, traditional retrieval methods like Entity Name Matching (ENM) are prone to noise and lack global context. We introduce Prototype-KRG, a novel framework that utilizes human-written "prototype" dialogues as a heuristic to navigate massive knowledge graphs. By identifying relevant commonsense facts from semantically similar past conversations, Prototype-KRG generates responses that are significantly more informative and diverse than existing state-of-the-art baselines.

Introduction

The primary challenge in knowledge-aware dialogue generation is knowledge selection. Standard approaches rely on ENM, which matches keywords in a user's query to entities in a KG. This method faces three critical limitations:

  1. Noise: Common words (e.g., "apple") can trigger hundreds of irrelevant facts.
  2. Locality: It focuses on individual words rather than the global context or "vibe" of the utterance.
  3. Depth: It typically only retrieves 1-hop neighbors, missing deeper semantic connections.

To address these, we propose using human-written dialogues as a natural filter, assuming that if a human used a specific fact to respond to a similar query in the past, that fact remains relevant for the current context.

Methods

The proposed system consists of two main components: a retrieval module (Prototype-KR) and a generation module (Prototype-KRG).

Prototype-KR: The Knowledge Filter

Instead of direct keyword matching, Prototype-KR searches a repository of human-written dialogues to find "prototypes"—past conversations semantically similar to the current query. It employs a three-stage ranking process (Retrieval → Coarse-grained ranking → Fine-grained ranking) using Jaccard similarity and sentence-level embeddings to select the top $k$ knowledge facts associated with those prototypes.

Prototype-KRG: The Generator

As shown in Figure 1, the generative model utilizes a Bi-GRU encoder and a specialized decoder. A Knowledge Bridge Fusion component integrates the query context with the selected knowledge facts to initialize the decoder's state. During generation, the decoder employs Mode Fusion, allowing it to choose between four modes: generating a standard vocabulary word, copying from the user query, or selecting an entity from either the prototype-retrieved knowledge or standard ENM facts.

Overview of the Prototype-KRG architecture, showing the integration of triple source knowledge and the knowledge-aware decoder.

Experimental Setup

The model was trained on a large-scale Chinese conversational dataset from Weibo (847K pairs) and grounded in ConceptNet, which contains 27K entities and 661K triples (e.g., [Apple, IsATypeOf, Fruit]).

Results and Discussion

Prototype-KRG was evaluated against several competitive baselines, including CCM (Commonsense Knowledge Aware Network) and ProtoEdit.

Quantitative Performance

As detailed in Table 1, our model significantly outperformed baselines in both knowledge utilization and response diversity. Specifically, the model achieved an EntN score of 1.43 (entities per response), compared to 0.92 for standard Seq2Seq and 1.13 for CCM. Furthermore, it showed a dramatic improvement in diversity, with a DIST-2 score of 20.32, nearly four times higher than CCM's 5.33.

Quantitative comparison of Prototype-KRG against baseline models across various metrics including EntN, diversity (DIST), and ROUGE.

Human Evaluation

Blind human evaluations confirmed these findings, with annotators preferring Prototype-KRG responses for both appropriateness and informativeness. The use of prototype dialogues allowed the model to understand global context; for example, recognizing that "My phone is broken" suggests knowledge about "buying a new iPhone" rather than just generic facts about "phones."

Conclusion

Prototype-KRG demonstrates that human-written dialogues serve as an effective heuristic for filtering large-scale knowledge graphs. By combining traditional keyword matching with prototype-based retrieval, the model achieves a superior balance of relevance and informativeness. Future work will explore simplifying the generation process and incorporating diverse knowledge types, such as persona-based or factual data.

Summary updated

BibTeX
@inproceedings{wu2020improving,
  author = {Sixing Wu and Ying Li and Dawei Zhang and Zhonghai Wu},
  title = {{Improving Knowledge-Aware Dialogue Response Generation by Using Human-Written Prototype Dialogues}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=9EZmgnFKWpU},
}
Figure