Trustworthy Conceptual Explanations for Neural Networks in Robot Decision-Making

Som Sagar, Aditya Taparia, Harsh Mankodiya, Pranav Bidare, Yifan Zhou, Ransalu Senanayake

TL;DR

Deep neural networks controlling robots are notoriously opaque, and standard explainable AI (XAI) methods often fail on continuous actions or offer ungrounded explanations without any measure of confidence. To solve this, the authors introduce BaTCAVe (Bayesian Testing with Concept Activation Vectors), a post-hoc diagnostic method that explains robot decision-making through human-interpretable concepts while providing principled uncertainty estimates to measure explanation trustworthiness.

What Problem Does This Paper Address?

While deep reinforcement learning and imitation learning enable robots to perform complex tasks, modern robotics still relies heavily on black-box policies. If a robot drops an object, turns off course, or halts unexpectedly, engineers and auditors need to understand why.

Existing XAI tools like Grad-CAM or LIME highlight image pixels or localized features, but they do not translate to high-level semantic concepts (e.g., "distance to obstacle" or "color of target"). Furthermore, existing concept-based explanation frameworks (such as TCAV) were built for discrete classification in computer vision, failing in two crucial ways when applied to robotics:

  1. They cannot readily handle continuous control signals or specific subsets of robot actions.
  2. They output point estimates without epistemic uncertainty, making it impossible to determine whether an explanation is reliable or merely a spurious artifact.

Methods

BaTCAVe extends Concept Activation Vectors (CAVs) into continuous robotic action spaces within a Bayesian framework:

  • Action and Input Concepts: The user defines an Action Concept ($C_A$) as a rule or condition over the output space (e.g., "top 25% of downward velocity" or "steering angle $> 2^\circ$") and probes candidate Input Concepts ($C_I$) represented by positive and negative exemplar data (e.g., images of dark vs. light surfaces, or proprioceptive state vectors).
  • Bayesian CAV Formulation: BaTCAVe treats the concept vector $v$ in internal activation space as a random variable. Because the Bernoulli likelihood of separating concept classes has a non-conjugate Gaussian prior, the posterior $q(v)$ is approximated using variational inference (via an Expectation-Maximization local linear approximation).
  • Stochastic Directional Derivatives: Sensitivity of the chosen action concept with respect to internal layer activations is evaluated across samples drawn from $q(v)$. This yields a concept attribution score ($s_{C_A, C_I, C_I^-}$) alongside an empirical standard deviation measuring epistemic uncertainty.
  • Trustworthiness Taxonomy: Explanations are categorized into three regimes based on concept classification accuracy and posterior uncertainty:
  • Off-base (untrustworthy): Low concept classification accuracy.
  • Imprecise: High accuracy but high uncertainty (e.g., conflicting concepts or lack of activation diversity).
  • Precise (trustworthy): High accuracy and low uncertainty.
+-----------------------------------------------------------------+
|                       The BaTCAVe Pipeline                      |
|                                                                 |
|   1. Define Action Concept (CA) & Input Concepts (CI)           |
|                                                                 |
|   2. Variational Inference on Activation Space                  |
|          Prior p(v) + Data (CI vs CI-) ---> Approx. q(v)        |
|                                                                 |
|   3. Stochastic Directional Derivative & Scoring                |
|          Sensitivity: d f(CA) / d f(-l)  *  v,  where v ~ q(v)  |
|                                                                 |
|   4. Output: Explanation Score + Epistemic Uncertainty          |
+-----------------------------------------------------------------+

Datasets

The authors evaluated BaTCAVe across four distinct robotic setups:

  • JetBot Obstacle Avoidance (Vision): Real-world robot with an onboard camera and NVIDIA Jetson Nano. Trained on 350 images (175 obstacle / 175 free space) and evaluated on 150 test images using custom concept sets (color swatches, HSV brightness, obstacle distances at 30cm, 45cm, and 60cm).
  • RoboSuite Manipulation Benchmark (Proprioception): Panda 6-DOF arm trained on the ph_low_dim offline demonstration dataset (robomimic) across three tasks: Lift Cube, Pick & Place Can, and Nut Assembly. Concepts were constructed by masking and perturbing low-dimensional proprioceptive vectors (EEF position, quaternion, gripper state, object pose).
  • Vision-Language Manipulation: A simulated cube-lifting task conditioning a ResNet/CLIP controller on paired image observations and language instructions. Concepts included segmented image components and language variants (verbs, full sentences, gibberish).
  • Donkey Simulator (End-to-End Driving): Vision-based autonomous steering trained via Proximal Policy Optimization (PPO). Concepts included track surface colors ("black shades") and obstacle cones.

Key Insights and Ideas

  • Catching Hidden Clever-Hans Shortcuts: When fine-tuning an AlexNet model to avoid an orange obstacle, human engineers assumed the network relied on the color orange. BaTCAVe revealed the network actually latched onto brightness/darkness in HSV space. Fine-tuning only the last layer (LFT) rather than full-network fine-tuning (FFT) helped steer the representation back toward the intended orange concept.
  • Isolating Policy Errors Across Layers: In the autonomous driving scenario, evaluating BaTCAVe on the final CNN layer yielded an explanation score of $\bar{s} = 0$, while the subsequent MLP layer produced $\bar{s} = 0.822$. This pinpointed that the perception backbone extracted appropriate road features, meaning navigation errors stemmed from downstream reinforcement learning policy optimization rather than visual encoder failure.
  • Temporal Disentanglement: In the vision-language robot arm task, language concepts revealed that the command's verb dominated policy attention at the precise moment of lifting, while nouns and whole sentences contributed marginally.

Key Results

  • Manipulation Sensitivity (Table I & II): In the Lift Cube task under downward end-effector movement ($\Delta Z \uparrow$), object pose ($0.98 \pm 0.10$) and end-effector position ($0.99 \pm 0.04$) showed near-maximal explanatory power with tight confidence bounds, whereas gripper status and orientation showed low influence with high uncertainty ($0.39 \pm 0.48$ and $0.42 \pm 0.49$).
  • Data Augmentation Stability: Last-layer fine-tuning (LFT) preserved stable concept attributions across color jitter and rotation augmentations, whereas full-network fine-tuning (FFT) exhibited erratic sensitivity shifts.
  • Autonomous Recovery: In Donkey Simulator, when the car strayed off-track at time $t = 35$, its BaTCAVe attribution to "black shades" (road) spiked at $t = 41$, confirming the recovery steering maneuver was driven by road-surface re-acquisition.

Limitations and Future Work

  • Small Concept Signal Weakness: When visual concepts occupy very few pixels (e.g., small, distant cones in the driving simulator), their latent representations become indistinguishable from random noise, reducing concept classifier accuracy.
  • Inability to Disentangle Ambiguity: Standard variational Bayesian inference cannot differentiate between a lack of diverse data versus genuine, conflicting concept signals in activation space.
  • Predefined Concept Reliance: BaTCAVe relies on predefined candidate concept datasets; future work aims to incorporate automated concept dictionary generation tailored for specific robotic tasks.

Summary updated

Abstract

Black box neural networks are an indispensable part of modern robots. Nevertheless, deploying such high-stakes systems in real-world scenarios poses significant challenges when the stakeholders, such as engineers and legislative bodies, lack insights into the neural networks’ decision-making process. Presently, explainable AI is primarily tailored to natural language processing and computer vision, falling short in two critical aspects when applied in robots: grounding in decision-making tasks and the ability to assess trustworthiness of their explanations. In this paper, we introduce a trustworthy explainable robotics technique based on human-interpretable, high-level concepts that attribute to the decisions made by the neural network. Our proposed technique provides explanations with associated uncertainty scores for arbitrary concepts using variational inference on a concept classifier within an explainable manifold. To validate our approach, we conducted a series of experiments with various simulated and real-world robot decision-making models, demonstrating the effectiveness of the proposed approach as a post-hoc, human-friendly robot learning diagnostic tool.
BibTeX
@inproceedings{sagar2025trustworthy,
  author = {Som Sagar and Aditya Taparia and Harsh Mankodiya and Pranav Bidare and Yifan Zhou and Ransalu Senanayake},
  title = {{Trustworthy Conceptual Explanations for Neural Networks in Robot Decision-Making}},
  booktitle = {RobotEvaluation @ RSS},
  year = {2025},
  url = {https://openreview.net/forum?id=nMBLpTy61c},
}

Loading related content...

Figure