Augmenting Knowledge-grounded Conversations with Sequential Knowledge Transition

Haolan Zhan, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Yongjun Bao, Yanyan LanNAACL 2021 · OpenReview

Abstract: Haolan Zhan, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Yongjun Bao, Yanyan Lan. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Introduction

Knowledge-grounded conversation (KGC) aims to enhance chatbots by providing access to external factual information, such as Wikipedia or structured databases. Despite progress, existing models often struggle with three key issues: static knowledge selection (ignoring conversational flow), overfitting (memorizing training data rather than generalizing), and knowledge neglect (failing to incorporate selected facts into the final response).

The SKT-KG (Sequential Knowledge Transition - Knowledge Grounded) model addresses these challenges by explicitly modeling how topics naturally shift during a conversation and ensuring the generator is conditioned to use external knowledge effectively.

The SKT-KG Framework

The SKT-KG model decomposes the response generation process into three distinct phases to improve coherence and factual relevance.

Overview of the SKT-KG architecture, showing the pre-training, fine-tuning, and sequential knowledge transition phases.
  1. Sequential Knowledge Transition (SKT): Instead of treating knowledge selection as an isolated event at each turn, SKT models it as a sequence. It abstracts specific facts into "topic tags" (e.g., "Aries" is tagged as astrological sign). A BiLSTM-CRF architecture predicts the next likely topic tag based on the dialogue history, capturing the "grammar" of topic shifts.
  2. Knowledge-Aware Pre-training: To combat knowledge neglect, the model undergoes a pre-training phase where it learns to transform a knowledge triple (Subject, Relation, Object) into a natural sentence. This forces the generator to prioritize the provided facts before it ever handles a full dialogue.
  3. Flexible Self-Attention Masking: A specialized masking strategy in the Transformer blocks allows the model to switch seamlessly between pre-training (knowledge-to-text) and fine-tuning (context + knowledge-to-response) without losing performance.

Knowledge as a Trajectory

A core insight of this work is that human conversation follows a path rather than a series of disconnected points. If a discussion begins with an actor's "astrological sign," it might naturally transition to their "blood type" or a "masterpiece" they starred in.

An example of a conversation and its corresponding knowledge transition path through topic tags.

By predicting high-level topic tags rather than raw text, the model generalizes better to unseen topics. It learns the underlying structure of how information is shared, making the resulting dialogue feel more focused and human-like.

Experimental Evaluation

The model was tested on two diverse benchmarks:

  • DuConv: A Chinese dataset featuring structured knowledge graph triples about movies and celebrities.
  • Wizard of Wikipedia (WoW): An English dataset using unstructured Wikipedia paragraphs, including a "Test Unseen" split to evaluate generalization.

Key Results

SKT-KG significantly outperformed state-of-the-art baselines like MemNet and PostKS across both datasets:

  • Performance: On DuConv, it achieved a BLEU-1 score of 37.80, surpassing the previous best sequential model, SLKS (33.93).
  • Generalization: On the WoW "Test Unseen" set, SKT-KG maintained superior BLEU and Diversity metrics, demonstrating its ability to handle novel topics it did not encounter during training.
  • Human Evaluation: In blind tests, human judges rated SKT-KG higher for relevance, informativeness, and naturalness. On DuConv, its relevance score was 2.3/3.0, compared to 1.9 for the SLKS baseline.

Conclusion and Future Directions

SKT-KG demonstrates that modeling the sequential nature of knowledge transitions is vital for building informative and coherent chatbots. While the model currently relies on external tools like LDA to extract topic tags for unstructured data, future work aims to use Variational Autoencoders (VAEs) to discover these latent topic transitions automatically. This would further enhance the model's flexibility across different domains and data formats.

Summary updated

BibTeX
@inproceedings{zhan2021augmenting,
  author = {Haolan Zhan and Hainan Zhang and Hongshen Chen and Zhuoye Ding and Yongjun Bao and Yanyan Lan},
  title = {{Augmenting Knowledge-grounded Conversations with Sequential Knowledge Transition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=xgGP1iaplES},
}
Figure