Explainable Knowledge Graph Embedding: Inference Reconciliation for Knowledge Inferences Supporting Robot Actions

Angel Daruna, Devleena Das, Sonia Chernova

TL;DR

This paper introduces an inference reconciliation framework that allows robots to explain the "black-box" knowledge inferences driving their actions. By using decision trees to locally approximate complex knowledge graph embeddings, the system generates natural language explanations that enable non-expert users to identify and correct a robot's nonsensical beliefs, leading to a 33.7% improvement in task success.

What Problem Does This Paper Address?

Modern robots often rely on Knowledge Graph Embeddings (KGEs) to handle ambiguity and unforeseen scenarios. For instance, if a robot is told to "fetch coffee," a KGE might help it infer that coffee is likely found in a kitchen. However, KGEs are essentially black boxes; their internal logic is a mess of high-dimensional vectors that are unintelligible to humans.

When a robot behaves erratically—like looking for a sponge in a microwave—a user needs to know why the robot thinks that makes sense. Existing Explainable AI (XAI) for robotics usually focuses on the robot's plan (e.g., "I'm moving to the sink to get a scrubber") rather than the underlying semantic knowledge (e.g., "I think sponges are found in microwaves"). This paper fills that gap by creating a way for robots to justify their knowledge-based inferences to non-expert users.

Methods

The authors propose a "pedagogical" XAI approach, where a simple, interpretable model is used to explain the behavior of a complex, black-box model.

  1. Knowledge Representation: The robot uses a standard KGE (specifically TuckER) to represent its world knowledge.
  2. Interpretable Graph Feature Model ($\Phi$): To explain a specific inference (a "triple" like sponge-LocIn-sink), the system extracts local features from the graph using Subgraph Feature Extraction (SFE).
  3. Local Approximation: Instead of trying to explain the entire KGE at once, the framework trains a Decision Tree Classifier to locally approximate the KGE's predictions for a specific query and its nearest neighbors in the embedding space.
  4. Natural Language Generation: The paths through the decision tree are grounded back into the original knowledge graph and converted into human-readable sentences using templates.
Fig. 1: Overview of our inference reconciliation framework which introduces a novel explainable knowledge graph embedding method (graph feature model), leveraging decision trees, to provide natural language explanations to users.

Datasets

The researchers used the VirtualHome dataset, which simulates household activities.

  • Scale: 291 entities (rooms, objects, actions, states) and thousands of triples across 11 relation types (e.g., ObjInRoom, HasEffect).
  • Noise Injection: To test the framework's ability to help users "debug" a robot, they created a corrupted version of the dataset by randomly flipping 30% of the facts to simulate sensor noise or incorrect learning.

Key Insights and Ideas

  • Knowledge vs. Causal Questions: The study found that users clearly distinguish between "Why are you doing that?" (causal) and "Why do you think that is true?" (knowledge). Causal explanations are better for plans, but knowledge-based explanations are essential when the robot's underlying beliefs are wrong.
  • Locality is Key: Trying to explain a global black-box model is hard. By focusing on the "local neighborhood" of a specific inference (using cosine similarity in the embedding space), the decision tree can achieve much higher fidelity to the black box.
  • Explanations as Debugging Tools: Explanations aren't just for building trust; they are actionable. When a robot explains a wrong inference, a human can provide feedback that directly improves the robot's future performance.

Key Results

  • High Fidelity: The decision tree surrogate model achieved an F1-Fidelity of 98.9%, significantly outperforming prior linear-model baselines (87.9%).
  • User Preference: In user studies, participants preferred the framework's knowledge-graph-based explanations for "why" questions about knowledge in 73.3% of analyzed interactions.
  • Performance Boost: When non-experts used the GUI to correct the robot's nonsensical beliefs (e.g., correcting "a living room contains a toothpaste tube" to "a bathroom contains a toothpaste tube"), the robot's link-prediction performance (MRR) jumped by 117%.
Graph showing how non-expert feedback accuracy correlates with improvements in the robot's link-prediction performance (MRR).
  • Task Success: Most importantly, this human-in-the-loop correction led to a 33.7% relative improvement in the robot's ability to successfully execute household tasks in simulated environments.

Limitations and Future Work

  • Template-Based Language: The current system uses templates for natural language, which can feel repetitive. Future work could integrate more sophisticated Large Language Models (LLMs) to generate more varied and natural responses.
  • Scalability: While effective for household graphs, the computational cost of local SFE and decision tree training for every query might need optimization for massive, web-scale knowledge graphs.
  • Real-World Noise: The study used simulated noise; testing the framework against actual robotic perception errors in physical environments is a logical next step.

Summary updated

Abstract

Learned knowledge graph representations supporting robots contain a wealth of domain knowledge that drives robot behavior. However, there does not exist an inference reconciliation framework that expresses how a knowledge graph representation affects a robot's sequential decision making. We use a pedagogical approach to explain the inferences of a learned, black-box knowledge graph representation, a knowledge graph embedding. Our interpretable model, uses a decision tree classifier to locally approximate the predictions of the black-box model, and provides natural language explanations interpretable by non-experts. Results from our algorithmic evaluation affirm our model design choices, and the results of our user studies with non-experts support the need for the proposed inference reconciliation framework. Critically, results from our simulated robot evaluation indicate that our explanations enable non-experts to correct erratic robot behaviors due to nonsensical beliefs within the black-box.
BibTeX
@inproceedings{daruna2022explainable,
  author = {Angel Daruna and Devleena Das and Sonia Chernova},
  title = {{Explainable Knowledge Graph Embedding: Inference Reconciliation for Knowledge Inferences Supporting Robot Actions}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9982104},
}
Figure