DyKgChat: Benchmarking Dialogue Generation Grounding on Dynamic Knowledge Graphs

Yi-Lin Tuan, Yun-Nung Chen, Hung-yi Lee

Abstract

Yi-Lin Tuan, Yun-Nung Chen, Hung-yi Lee. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Most "knowledge-grounded" conversation models rely on static knowledge bases—fixed sets of facts like "Paris is the capital of France." However, real-world information is often temporal and evolving. This paper introduces DyKgChat, a new benchmark and dataset for dialogue systems that must ground their responses in dynamic knowledge graphs (KGs) that evolve over time. The authors also propose Qadpt, a model that uses multi-hop reasoning over graph structures to achieve "zero-shot adaptation," allowing it to update its answers immediately when facts in the underlying KG change without requiring any retraining.

The Challenge of Shifting Facts

In dynamic environments like TV shows, business, or news, relationships and facts change constantly. Current neural architectures struggle with two primary issues in this context:

  1. Zero-shot Adaptation: Standard models usually require retraining or fine-tuning to stop generating outdated information when a fact changes (e.g., Character A and B were friends but are now enemies).
  2. Multi-hop Reasoning: Many responses require connecting multiple dots (e.g., "Where does your friend's brother live?"). Existing models often only look at direct "one-hop" relationships.
Example of dynamic knowledge grounding and zero-shot adaptation in the DyKgChat framework.

Methods: The Qadpt Model

The researchers propose Qadpt (Quick Adaptative Dynamic Knowledge-Grounded Model). Its architecture is designed to decouple knowledge from model weights, ensuring that the knowledge remains modular and swappable.

Architecture Components

  • Seq2Seq Backbone: A standard Gated Recurrent Unit (GRU) encoder-decoder handles the general flow of the conversation.
  • The Controller: A gating mechanism decides at each time step whether the model should output a "generic" word (like "the" or "hello") or a "knowledge entity" from the KG.
  • The Reasoning Model: This is the core innovation. Instead of using static graph embeddings, the model explicitly factorizes a transition matrix ($T_t$) based on the KG's adjacency matrix ($A$). By treating the KG as a Markov chain and performing $N$ steps of matrix multiplication, the model can "walk" the graph to find entities several hops away.
The Qadpt reasoning model architecture, illustrating how the transition matrix enables multi-hop reasoning over the knowledge graph.

Because the reasoning is tied directly to the adjacency matrix, if a single edge in the KG is changed, the model’s output distribution changes instantly—enabling zero-shot adaptation without retraining.

The DyKgChat Dataset

The authors introduced the DyKgChat corpus, derived from two popular TV series:

  1. HGZHZ (Hou Gong Zhen Huan Zhuan): A Chinese palace drama with 76 episodes, featuring complex, shifting social hierarchies (~17k turns).
  2. Friends: The classic English sitcom (~57k turns).

Unlike many distantly labeled datasets, these KGs were manually constructed from fan wikis to ensure accuracy. Data analysis revealed that in these dramas, most shortest paths between entities mentioned in a prompt and the correct response are 2 to 3 hops long, making multi-hop reasoning a mathematical necessity for solving these queries.

Key Results

The models were evaluated on Accurate Change Rate—how often the model correctly updates its output when the KG is modified.

  • Superior Adaptation: On the HGZHZ dataset, when a relation was changed, the baseline MemNet model only adapted correctly ~1.17% of the time. In contrast, the proposed Qadpt model achieved a 66.67% accurate change rate.
  • Knowledge Accuracy: Qadpt outperformed baselines in "Knowledge Word Accuracy" (KW-Acc), proving it was better at picking the right facts even in static scenarios.
  • Human Evaluation: Human judges found Qadpt's responses to be more informative and fluent than standard Seq2Seq and memory-network-based baselines.

Conclusion and Future Work

By keeping the KG as an external adjacency matrix and using the neural network only to predict the path (the sequence of relations), Qadpt achieves a level of flexibility previously unseen in grounded dialogue systems. Future work will focus on scaling these methods to massive, real-world KGs like Wikidata and exploring continuous evolution during long-term dialogues.

Summary updated

BibTeX
@inproceedings{tuan2019dykgchat,
  author = {Yi-Lin Tuan and Yun-Nung Chen and Hung-yi Lee},
  title = {{DyKgChat: Benchmarking Dialogue Generation Grounding on Dynamic Knowledge Graphs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=I3teu9oQkA},
}
Figure