TL;DR
This paper introduces BaTCAVe (Bayesian Testing with Concept Activation Vectors), a post-hoc explainability framework designed specifically for robotic deep neural networks. By mapping network activations to high-level, human-interpretable concepts and quantifying the epistemic uncertainty of these mappings, BaTCAVe provides trustworthy, grounded explanations for continuous robot behaviors.
What Problem Does This Paper Address?
Deep Neural Networks (DNNs) are incredibly powerful, but they remain "black boxes." In high-stakes robotics applications like autonomous driving or industrial manipulation, deploying an uninterpretable model is a massive safety and regulatory risk.
While Explainable AI (XAI) has flourished in computer vision and natural language processing, existing methods fall short in robotics for two main reasons:
- Lack of grounding in decision-making: Standard XAI methods (like Grad-CAM or LIME) highlight pixel regions or tokens, which doesn't translate well to continuous control signals or explain why a specific action sequence was chosen.
- No measure of trustworthiness: Current explainers will always output an explanation, even if it's completely wrong or arbitrary. In robotics, engineers need to know how much to trust the explanation itself.
Methods
BaTCAVe tackles these challenges by extending Concept Activation Vectors (CAVs)—traditionally used to find human concepts like "stripes" in image classifiers—into a probabilistic framework suitable for robotics.
The methodology consists of three core pillars:
- Action Concepts ($C_A$): Instead of explaining an entire trajectory at once, users define specific target behaviors using logical rules over the robot's action space (e.g., "when the end-effector moves down while the gripper is open").
- Input Concepts ($C_I$): High-level semantic attributes (e.g., "redness," "cube shape," "distance") represented by a small dictionary of example inputs.
- Bayesian Linear Separation: BaTCAVe probes an intermediate layer of the robot's policy network. It uses Variational Inference to learn a probability distribution over the hyperplanes that separate positive and negative concept examples in the activation space.
By computing the directional derivative of the action policy along these probabilistic concept vectors, BaTCAVe generates an explanation score distribution, yielding both a mean relevance score and an epistemic uncertainty score.
Datasets
The authors evaluated BaTCAVe across four distinct robotic setups:
- Vision-based Mobile Robot Navigation (Real-world JetBot): A custom dataset consisting of 350 training and 150 testing images of an orange box obstacle vs. free space.
- Proprioceptive Manipulation (RoboSuite): Low-dimensional sensor datasets from the robomimic benchmark covering three tasks: Lift Cube, Pick & Place, and Nut Assembly.
- Vision-Language Manipulation: A complex simulation environment where a robot arm receives high-resolution RGB images alongside natural language instructions, utilizing pre-trained ResNet and CLIP encoders.
- Autonomous Driving (Donkey Simulator): An end-to-end vision-based driving policy trained via Proximal Policy Optimization (PPO).
Key Insights and Ideas
- Human Intuition Can Be Wrong: In the JetBot navigation task, 35% of surveyed engineers guessed the robot avoided the obstacle because it learned "orangeness." BaTCAVe revealed the network actually learned to distinguish "dark" from "light" objects; the orange box was simply treated as a shade of "dark."
- Uncertainty Categorization: The authors categorize explanations into three distinct trust profiles based on accuracy and uncertainty:
- Off-base: Low accuracy (concepts don't separate well; do not trust).
- Imprecise: High accuracy but high uncertainty (multiple explanations possible due to conflicting info or non-diverse test data).
- Precise: High accuracy and low uncertainty (highly trustworthy explanation).
- Debugging Network Layers: In the autonomous driving experiment, applying BaTCAVe to different layers revealed that steering errors did not originate in the CNN image encoder (which successfully tracked "road" concepts) but rather in the subsequent MLP policy layers, telling engineers exactly where more training was needed.
Key Results
- Proprioceptive Tasks: In RoboSuite manipulation tasks, BaTCAVe successfully identified that object features were dominant for horizontal alignment ($\Delta X$ score: $1.00 \pm 0.00$), while gripper status became highly relevant only when vertical movement ($\Delta Z$) was initiated.
- Vision-Language Grounding: For the vision-language cube-lifting task, BaTCAVe demonstrated that the policy network heavily prioritized the verb in a command over nouns or gibberish precisely at the moment of lifting.
- Robustness to Augmentation: The framework proved that fine-tuning only the final layer (Layer-Wise Fine-Tuning) preserved the target concept ("orangeness") much better under data augmentations like color jittering compared to Full-Network Fine-Tuning.
Limitations and Future Work
The primary limitation identified is signal dilution. When a concept is physically small within the input space (such as traffic cones that are far away in the driving simulator), it fails to generate a strong enough activation signal to be cleanly separated from random noise.
The authors note that future work will focus on expanding BaTCAVe to explain long-horizon sequential decisions rather than isolated behaviors, and developing activation-amplification methods to better capture small or distant environmental cues.
Summary updated