Abstract
Black box neural networks are an indispensable part of modern robots. Nevertheless, deploying such high-stakes systems in real-world scenarios poses significant challenges when the stakeholders, such as engineers and legislative bodies, lack insights into the neural networks' decision-making process. Presently, explainable AI is primarily tailored to natural language processing and computer vision, falling short in two critical aspects when applied in robots: grounding in decision-making tasks and the ability to assess trustworthiness of their explanations. In this paper, we introduce a trustworthy explainable robotics technique based on human-interpretable, high-level concepts that attribute to the decisions made by the neural network. Our proposed technique provides explanations with associated uncertainty scores by matching neural network's activations with human-interpretable visualizations. To validate our approach, we conducted a series of experiments with various simulated and real-world robot decision-making models, demonstrating the effectiveness of the proposed approach as a post-hoc, human-friendly robot learning diagnostic tool.
Introduction
Deep Neural Networks (DNNs) have become central to robot control, yet their "black box" nature remains a significant barrier to deployment in high-stakes environments like autonomous driving or industrial collaboration. While traditional Explainable AI (XAI) methods such as Grad-CAM provide saliency maps, these heatmaps often fail to capture the semantic nuances required for robotics. For instance, a heatmap might indicate a robot is attending to an object, but it cannot distinguish if the decision is based on the object’s color, distance, or velocity.
To address these limitations, we present BaTCAVe (Bayesian Testing with Concept Activation Vectors). BaTCAVe maps high-dimensional robot activations to human-understandable concepts and, crucially, provides a built-in "trust score" via Bayesian uncertainty. This allows engineers to diagnose robot behavior and assess the reliability of the provided explanation.
Methodology
BaTCAVe builds upon the Testing with Concept Activation Vectors (TCAV) technique, which identifies directions in a neural network's internal activation space corresponding to human concepts. Our framework introduces three core innovations:
- Action Concepts: Users can define specific behaviors of interest (e.g., "Why did the robot turn right?") rather than explaining the entire model globally.
- Bayesian Inference: By employing Variational Inference, BaTCAVe treats explanations as probability distributions. This accounts for epistemic uncertainty, allowing the system to signal when data is insufficient for a reliable explanation.
- Trustworthiness Categorization: Explanations are classified into three distinct cases based on accuracy and uncertainty.
- Case 1 (Off-base): Low accuracy; the concepts do not explain the behavior.
- Case 2 (Imprecise): High accuracy but high uncertainty; multiple explanations may be valid.
- Case 3 (Precise): High accuracy and low uncertainty; a highly trustworthy explanation.
Experimental Evaluation
We validated BaTCAVe across four robotic domains: vision-based navigation (JetBot), proprioceptive manipulation (robomimic/Panda arm), vision-language tasks, and autonomous driving (Donkey Simulator).
Key Insights
- Correcting Human Intuition: In navigation tasks, engineers often assumed a robot avoided obstacles based on color (orange). BaTCAVe revealed the model actually learned the concept of "darkness," as the obstacles were darker than the floor. This highlights how XAI can catch "clever Hans" shortcuts that humans overlook.
- Diagnostic Precision: In autonomous driving, BaTCAVe identified that errors originated in the policy layers (MLP) rather than the visual encoder (CNN), directing engineering efforts toward decision-making logic.
- Temporal Alignment: In vision-language tasks, the framework demonstrated that the "verb" concept (e.g., "Lift") becomes dominant precisely when the physical action begins, validating multi-modal alignment.
Quantitative Results
BaTCAVe provided actionable insights where traditional methods failed. In manipulation tasks, it correctly identified "object features" as the primary driver for Pick & Place tasks ($\Delta X$ score of $1.00 \pm 0.00$), while "gripper status" was identified as critical for Nut Assembly. The Bayesian approach successfully filtered "imprecise" explanations that standard TCAV would have presented with false confidence.
Discussion and Future Work
While BaTCAVe offers significant improvements in transparency, it faces limitations regarding signal strength for physically small concepts in the input. Future research will focus on the automated discovery of concepts from robot experience and extending the framework to explain long-term sequential strategies and complex multi-step trajectories.
Summary updated