More is Better: Enhancing Open-Domain Dialogue Generation via Multi-Source Heterogeneous Knowledge

Sixing Wu, Ying Li, Minghui Wang, Dawei Zhang, Yang Zhou, Zhonghai WuEMNLP 2021 · OpenReview

Abstract

Sixing Wu, Ying Li, Minghui Wang, Dawei Zhang, Yang Zhou, Zhonghai Wu. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

2021.

Introduction

Open-domain dialogue systems, or chatbots, frequently encounter the "long-tail" problem: while they are designed to converse on any topic, their internal knowledge is often insufficient for specific factual queries. Knowledge-grounded generation attempts to mitigate this by retrieving external facts. However, most current models rely on a single, homogeneous knowledge source, such as a knowledge graph or a text corpus. This reliance leads to limited coverage and "Topic Conflict," where retrieved facts may not align with the conversational context.

Methodology: The MSKE-Dialog Model

To address these limitations, the authors introduce MSKE-Dialog, a framework that integrates four encoders to process dialogue context alongside three heterogeneous knowledge sources: commonsense triplets, plain text, and structured infoboxes.

The MSKE-Dialog architecture, featuring multi-source encoders and specialized selection/generation gates.

Multi-Reference Selection

The model employs a two-tier filtering mechanism to ensure relevance. A Global Relevance Gate assesses which knowledge source is most appropriate for a given query, while a Dynamic Selection Gate performs fine-grained selection of specific entries at each decoding step.

Multi-Reference Generation

To resolve "Generation Conflict"—where the same word might appear in different data distributions across sources—MSKE-Dialog constructs a Unified Dynamic Vocabulary. This merges tokens from various sources into a single space. Additionally, a Dynamic Copy Mechanism allows the model to directly incorporate specific entities (e.g., product names) from the retrieved knowledge into the generated response.

Experimental Setup

The model was evaluated using a large-scale Chinese Weibo dataset containing 700,000 training pairs. Knowledge was sourced from ConceptNet (commonsense) and Chinese Wikipedia (text and infoboxes). Alignment efforts ensured that approximately 80% of dialogues were linked to external knowledge, significantly increasing coverage compared to single-source baselines.

Results and Analysis

MSKE-Dialog was compared against ten state-of-the-art baselines. Automatic metrics revealed that the model outperformed competitors in 12 out of 15 categories, notably in BLEU (accuracy), DIST (diversity), and Entity Scores.

Comparison of MSKE-Dialog against state-of-the-art baselines across various automatic metrics.

Key Findings

  • Synergy of Heterogeneity: Combining different knowledge types compensates for individual weaknesses; for instance, the logical structure of knowledge graphs complements the rich information in plain text.
  • Efficiency vs. Scale: MSKE-Dialog (59M parameters) outperformed larger models like CDial-GPT (95M parameters), suggesting that structured knowledge integration is more effective than simply increasing parameter count.
  • Robustness: The model maintained high performance even with reduced training data (25%), as external knowledge provided a necessary "crutch" for generation.

Human evaluations corroborated these findings, with judges rating MSKE-Dialog's responses as more appropriate and informative in the majority of head-to-head matchups.

Conclusion and Future Work

While MSKE-Dialog sets a new benchmark for knowledge-grounded dialogue, limitations remain in decoding strategies and the quality of initial entity retrieval. Future work will explore integrating multi-source knowledge with massive-scale pre-trained models like GPT-3 to further enhance conversational depth and accuracy.

Summary updated

BibTeX
@inproceedings{wu2021more,
  author = {Sixing Wu and Ying Li and Minghui Wang and Dawei Zhang and Yang Zhou and Zhonghai Wu},
  title = {{More is Better: Enhancing Open-Domain Dialogue Generation via Multi-Source Heterogeneous Knowledge}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=8ZOa-RP0Bf},
}
Figure