BaTCAVe: Trustworthy Explanations for Robot Behaviors

Som Sagar, Aditya Taparia, Harsh Mankodiya, Pranav Bidare, Yifan Zhou, Ransalu Senanayake

TL;DR

This paper introduces BaTCAVe (Bayesian Testing with Concept Activation Vectors), a post-hoc explainability framework designed specifically for robotic deep neural networks. By mapping network activations to high-level, human-interpretable concepts and quantifying the epistemic uncertainty of these mappings, BaTCAVe provides trustworthy, grounded explanations for continuous robot behaviors.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are incredibly powerful, but they remain "black boxes." In high-stakes robotics applications like autonomous driving or industrial manipulation, deploying an uninterpretable model is a massive safety and regulatory risk.

While Explainable AI (XAI) has flourished in computer vision and natural language processing, existing methods fall short in robotics for two main reasons:

  1. Lack of grounding in decision-making: Standard XAI methods (like Grad-CAM or LIME) highlight pixel regions or tokens, which doesn't translate well to continuous control signals or explain why a specific action sequence was chosen.
  2. No measure of trustworthiness: Current explainers will always output an explanation, even if it's completely wrong or arbitrary. In robotics, engineers need to know how much to trust the explanation itself.

Methods

BaTCAVe tackles these challenges by extending Concept Activation Vectors (CAVs)—traditionally used to find human concepts like "stripes" in image classifiers—into a probabilistic framework suitable for robotics.

Fig. 1: In this pick-and-place task, users can request a post- hoc explanation for why the robot succeeded (or failed). BaTCAVe probes the policy network to obtain a ranked list of possible explanations, each with an associated likelihood score. In this example, the object’s redness and cylindrical shape are likely contributors to the robot’s actions. The uncertainty intervals indicate how much the user should trust each explanation. Explanations help with model debugging, auditing for regulatory compliance, and building trust.

The methodology consists of three core pillars:

  • Action Concepts ($C_A$): Instead of explaining an entire trajectory at once, users define specific target behaviors using logical rules over the robot's action space (e.g., "when the end-effector moves down while the gripper is open").
  • Input Concepts ($C_I$): High-level semantic attributes (e.g., "redness," "cube shape," "distance") represented by a small dictionary of example inputs.
  • Bayesian Linear Separation: BaTCAVe probes an intermediate layer of the robot's policy network. It uses Variational Inference to learn a probability distribution over the hyperplanes that separate positive and negative concept examples in the activation space.

By computing the directional derivative of the action policy along these probabilistic concept vectors, BaTCAVe generates an explanation score distribution, yielding both a mean relevance score and an epistemic uncertainty score.

Fig. 2: The user specifies the robot behavior of interest for analysis using action concepts, CA (e.g., when the robot is acting near the table). The user also provides input concepts, CI, (e.g., potential explanations represented as a dictionary of images or proprioception data), and test data, x (e.g., images or proprioception data). BaTCAVe measures the similarity of activation strengths and directions between each set of input concepts and test data for the given behavior. To do this, the input concepts are linearly separated in the activation space using a Bayesian classifier, resulting in infinite number of classification boundaries, each with a different probability.

Datasets

The authors evaluated BaTCAVe across four distinct robotic setups:

  1. Vision-based Mobile Robot Navigation (Real-world JetBot): A custom dataset consisting of 350 training and 150 testing images of an orange box obstacle vs. free space.
  2. Proprioceptive Manipulation (RoboSuite): Low-dimensional sensor datasets from the robomimic benchmark covering three tasks: Lift Cube, Pick & Place, and Nut Assembly.
  3. Vision-Language Manipulation: A complex simulation environment where a robot arm receives high-resolution RGB images alongside natural language instructions, utilizing pre-trained ResNet and CLIP encoders.
  4. Autonomous Driving (Donkey Simulator): An end-to-end vision-based driving policy trained via Proximal Policy Optimization (PPO).

Key Insights and Ideas

  • Human Intuition Can Be Wrong: In the JetBot navigation task, 35% of surveyed engineers guessed the robot avoided the obstacle because it learned "orangeness." BaTCAVe revealed the network actually learned to distinguish "dark" from "light" objects; the orange box was simply treated as a shade of "dark."
  • Uncertainty Categorization: The authors categorize explanations into three distinct trust profiles based on accuracy and uncertainty:
  • Off-base: Low accuracy (concepts don't separate well; do not trust).
  • Imprecise: High accuracy but high uncertainty (multiple explanations possible due to conflicting info or non-diverse test data).
  • Precise: High accuracy and low uncertainty (highly trustworthy explanation).
Three cases of uncertainty when separating classes in the activation space.
  • Debugging Network Layers: In the autonomous driving experiment, applying BaTCAVe to different layers revealed that steering errors did not originate in the CNN image encoder (which successfully tracked "road" concepts) but rather in the subsequent MLP policy layers, telling engineers exactly where more training was needed.

Key Results

  • Proprioceptive Tasks: In RoboSuite manipulation tasks, BaTCAVe successfully identified that object features were dominant for horizontal alignment ($\Delta X$ score: $1.00 \pm 0.00$), while gripper status became highly relevant only when vertical movement ($\Delta Z$) was initiated.
  • Vision-Language Grounding: For the vision-language cube-lifting task, BaTCAVe demonstrated that the policy network heavily prioritized the verb in a command over nouns or gibberish precisely at the moment of lifting.
  • Robustness to Augmentation: The framework proved that fine-tuning only the final layer (Layer-Wise Fine-Tuning) preserved the target concept ("orangeness") much better under data augmentations like color jittering compared to Full-Network Fine-Tuning.

Limitations and Future Work

The primary limitation identified is signal dilution. When a concept is physically small within the input space (such as traffic cones that are far away in the driving simulator), it fails to generate a strong enough activation signal to be cleanly separated from random noise.

The authors note that future work will focus on expanding BaTCAVe to explain long-horizon sequential decisions rather than isolated behaviors, and developing activation-amplification methods to better capture small or distant environmental cues.

Summary updated

Abstract

Black box neural networks are an indispensable part of modern robots. Nevertheless, deploying such high-stakes systems in real-world scenarios poses significant challenges when the stakeholders, such as engineers and legislative bodies, lack insights into the neural networks' decision-making process. Presently, explainable AI is primarily tailored to natural language processing and computer vision, falling short in two critical aspects when applied in robots: grounding in decision-making tasks and the ability to assess trustworthiness of their explanations. In this paper, we introduce a trustworthy explainable robotics technique based on human-interpretable, high-level concepts that attribute to the decisions made by the neural network. Our proposed technique provides explanations with associated uncertainty scores for the explanation by matching neural network's activations with human-interpretable visualizations. To validate our approach, we conducted a series of experiments with various simulated and real-world robot decision-making models, demonstrating the effectiveness of the proposed approach as a post-hoc, human-friendly robot diagnostic tool.
BibTeX
@inproceedings{sagar2025batcave,
  author = {Som Sagar and Aditya Taparia and Harsh Mankodiya and Pranav Bidare and Yifan Zhou and Ransalu Senanayake},
  title = {{BaTCAVe: Trustworthy Explanations for Robot Behaviors}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11245893},
}
Figure