KG-CRuSE: Recurrent Walks over Knowledge Graph for Explainable Conversation Reasoning using Semantic Embeddings

Rajdeep Sarkar, Mihael Arcan, John P. McCraeACL 2022

TL;DR

This paper introduces KG-CRuSE, a novel LSTM-based model that leverages Sentence-BERT embeddings to generate explainable paths over knowledge graphs for conversational AI. By capturing rich semantic information from dialogue history and knowledge graph elements, KG-CRuSE significantly improves the accuracy and explainability of dialogue systems, outperforming state-of-the-art methods on the OpenDialKG dataset.

What Problem Does This Paper Address?

Modern dialogue systems, while capable of generating fluent responses, often lack factual grounding, leading to bland and uninformative conversations. Knowledge-grounded dialogue systems aim to address this by incorporating external knowledge, typically from knowledge graphs (KGs). However, a key challenge is selecting the right facts from a KG that are relevant to the current dialogue context for response generation. This selection process is crucial for generating informative and coherent responses, and importantly, for providing an explanation of why a particular piece of information was chosen. Existing methods for KG-based dialogue often struggle with effectively capturing the semantic nuances of both the dialogue and the KG, and their explainability is limited. This paper tackles the problem of generating explainable KG paths that are semantically relevant to the dialogue context, thereby improving both the quality and transparency of knowledge-grounded dialogue systems.

Methods

The core of KG-CRuSE is its ability to generate "walks" or paths over a knowledge graph, guided by the dialogue history. The novelty lies in how it represents and processes this information:

  • Semantic Embeddings: Instead of relying on standard BERT embeddings, KG-CRuSE utilizes Sentence-BERT (SBERT) to encode both the dialogue history and KG elements (entities and relations). SBERT is chosen for its superior ability to capture sentence-level semantic similarity, which is crucial for aligning dialogue context with KG facts.
  • LSTM-based Recurrent Walks: An LSTM network models the temporal information of the dialogue and the path history. At each step, the LSTM takes the dialogue context and previously traversed KG elements as input, generating a hidden state. This hidden state is then used to predict the next action (relation and entity) to traverse in the KG.
  • Path Sampling: To handle the long-tailed distribution of KG neighbors and encourage exploration, KG-CRuSE samples the top-k semantically similar neighbors at each step, allowing for more diverse path exploration.
  • Explainability: The generated paths themselves serve as explanations, showing the sequence of facts and relations the model traversed to arrive at a particular piece of information.

Datasets

The primary dataset used for training and evaluation is OpenDialKG. This dataset contains dialogues annotated with KG paths, allowing for the evaluation of KG path generation and response generation. The authors split the dataset into 70% for training, 15% for validation, and 15% for testing.

Key Insights and Ideas

  • SBERT is Key: The choice of SBERT for encoding both dialogue and KG elements is a significant contributor to KG-CRuSE's performance. It effectively bridges the semantic gap between natural language dialogue and structured KG data.
  • Path Exploration Matters: Sampling multiple paths at each step (especially in the initial timesteps) allows the model to explore a wider range of relevant information, leading to more optimal path selection.
  • Explainability as a Feature: The generated paths are not just a means to an end (response generation) but are inherently valuable as explanations, making the dialogue system more transparent.
  • Fine-tuning Trade-offs: Fine-tuning the SBERT architecture for dialogue history encoding can sometimes decrease performance, suggesting that pre-trained SBERT embeddings already capture sufficient semantic information without overfitting to the specific dialogue domain.

Key Results

KG-CRuSE demonstrates strong performance across multiple metrics on the OpenDialKG dataset:

  • Quantitative Performance: KG-CRuSE outperforms several baseline models, including Tri-LSTM, Ext-ED, Seq2Path, Seq2Seq, and DKGW, on various recall@k metrics for entity and path retrieval. For instance, it achieves a path@1 accuracy of 19.59% and a tgt@1 accuracy of 20.20%, showing a relative improvement of at least 10% over baselines as 'k' increases.
  • Efficiency: While not the fastest in terms of training time per epoch compared to some models like DKGW, KG-CRuSE offers a significantly better balance of performance and inference speed compared to models like AttnIO, which has similar accuracy but much higher computational cost.
  • Qualitative Examples: The paper includes examples where KG-CRuSE generates paths that are semantically relevant and admissible, even if they differ from the gold-standard paths, highlighting its ability to find reasonable explanations.

Limitations and Future Work

  • Limited Dialogue Context: In some cases, KG-CRuSE might generate paths that are not sufficient to fully answer a user's query due to limited dialogue context.
  • Path Divergence: While KG-CRuSE can identify relevant paths, these might differ from the gold-standard paths, especially when the dialogue context is ambiguous. The authors suggest that this is acceptable as long as the paths are admissible and fit the context.
  • Future Work: The authors plan to extend the model to scenarios where ground truth paths are not available, focusing on generating knowledge-conditioned dialogues without explicit path annotations. They also aim to explore scenarios involving the generation of knowledge-conditioned dialogue.

Summary updated

BibTeX
@inproceedings{sarkar2022kg,
  author = {Rajdeep Sarkar and Mihael Arcan and John P. McCrae},
  title = {{KG-CRuSE: Recurrent Walks over Knowledge Graph for Explainable Conversation Reasoning using Semantic Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure