KSAM: Infusing Multi-Source Knowledge into Dialogue Generation via Knowledge Source Aware Multi-Head Decoding

Sixing Wu, Ying Li, Dawei Zhang, Zhonghai WuACL 2022 · OpenReview

Introduction

To enhance the human-likeness of AI conversations, researchers often infuse models with external knowledge. However, integrating multiple heterogeneous knowledge sources—such as unstructured text, structured tables, and commonsense knowledge graphs—presents significant challenges. Existing models often struggle with knowledge diversity, where different data formats cause interference in a single decoder; indefinite alignment, where the model fails to select the relevant source for a specific dialogue turn; and scalability, as adding new sources typically requires retraining the entire system.

Methodology: Multi-Head Decoding

The Knowledge Source Aware Multi-Head Decoding (KSAM) framework addresses these issues by decoupling the decoding process. Instead of a monolithic decoder, KSAM employs a multi-head architecture that treats each knowledge source as a distinct stream of information.

The KSAM architecture featuring source-specific encoders, multi-head decoders, and a source fusion network.

Source-Specific Encoders

The model utilizes specialized encoders for different input types: a Bi-GRU for dialogue history and independent Transformer-based encoders for plain text ($K_P$), commonsense facts ($K_C$), and structured tables ($K_T$).

Source-Aware Decoder Heads

Each knowledge source is assigned a dedicated, independent decoder head. Each head generates its own probability distribution for the next token based solely on its respective knowledge source. This prevents "knowledge interference" and allows for modular scalability, as new sources can be added without re-engineering the generative core.

Source Fusion Network (SFN)

The SFN acts as a conductor, merging the outputs from all heads. It employs two gating mechanisms: decoder head gates to determine the general utility of each source for the current context, and step-wise gates to dynamically adjust the importance of each source for every generated token.

Experimental Setup

KSAM was evaluated on a large-scale Chinese multi-source knowledge-aligned dataset containing approximately 70k training dialogues. The knowledge base included 696k commonsense facts from ConceptNet, 1.6M Wikipedia paragraphs, and 1.5M Wikipedia infobox tables. Nearly 80% of the dialogues were aligned with at least one knowledge type.

Results and Analysis

KSAM demonstrated superior performance across various automatic metrics compared to state-of-the-art baselines like MSKE.

Comparison of KSAM against baselines across automatic metrics including PPL, BLEU, and diversity.

As shown in the results table, KSAM achieved a significantly lower Perplexity (PPL) of 77.65, compared to MSKE's 81.10. It also showed improvements in BLEU-4 (2.15 vs 1.92) and DIST-2 (30.20 vs 27.50), suggesting more accurate and diverse responses. Human evaluations further confirmed these findings, with native speakers preferring KSAM for its rationality and informativeness in 54.2% of cases compared to MSKE.

Conclusion

KSAM introduces a flexible and scalable approach to multi-source knowledge infusion in dialogue systems. By utilizing independent decoder heads and a sophisticated fusion network, it effectively manages diverse information types and resolves alignment issues. Future work will focus on improving the fusion of "copy" mechanisms for better handling of out-of-vocabulary words and exploring more advanced decoding algorithms.

Summary updated

BibTeX
@inproceedings{wu2022ksam,
  author = {Sixing Wu and Ying Li and Dawei Zhang and Zhonghai Wu},
  title = {{KSAM: Infusing Multi-Source Knowledge into Dialogue Generation via Knowledge Source Aware Multi-Head Decoding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=nmZF0CLlppy},
}
Figure