Brittle interpretations: The Vulnerability of TCAV and Other Concept-based Explainability Tools to Adversarial Attack

Davis Brown, Henry Kvinge

TL;DR

Concept-based explainability tools like TCAV and Faceted Feature Visualization (FFV) help humans understand deep networks by mapping internal activations to user-provided concept examples ("tokens"). This paper introduces Token Pushing (TP) attacks, demonstrating that applying imperceptible perturbations to these centralized concept tokens can manipulate the explanations—fooling TCAV into reporting that stripes are irrelevant for identifying zebras, and breaking neuron visualizations without changing the underlying model.

What Problem Does This Paper Address?

Deep neural networks are often deployed in safety-critical domains like healthcare and finance, where interpretability is essential for auditing decisions. Concept-Based Interpretability Methods (CBIMs) interrogate models using collections of positive examples (concept tokens) representing human-understandable concepts (e.g., "stripes," "scales," or "textures").

While adversarial attacks on model predictions and saliency maps are well-documented, the security of CBIMs has remained largely unexamined. Because concept datasets are often curated centrally and reused across multiple audit pipelines, they introduce a single point of failure: what happens if an adversary tampers with the concept tokens themselves?

Methods

The authors formulate a formal threat model for post-hoc CBIMs and propose the Token Pushing (TP) attack:

  • Threat Model: The attacker has read/write access to the positive concept tokens ($P_C$) and read access to at least a surrogate model, but does not need access to the target dataset/neuron, nor do they need to know which specific interpretability algorithm will be run.
  • Objective Formulation: Instead of optimizing against a specific explainability metric, the attack disrupts the underlying geometric assumption of CBIMs—that semantically related inputs cluster closely in intermediate activation space.
  • Centroid PGD Attack: For a target layer $\ell$, the attacker calculates the centroid $\mu_T$ of representations of an unrelated image set $U_C$. Then, using Projected Gradient Descent (PGD with $\ell_\infty$ budget $\epsilon = 4/255$ over 20 iterations), each positive concept token $x_i^P \in P_C$ is perturbed to minimize its distance to $\mu_T$: $$\delta_i := \arg\min_{\|\delta\|_\infty \le \epsilon} \|f_\ell(x_i^P + \delta) - \mu_T\|$$
  • Target Explainability Methods:
  • TCAV (Testing with Concept Activation Vectors): Tests whether a concept vector aligns with the directional derivative of model predictions.
  • Faceted Feature Visualization (FFV): Disambiguates polysemantic neurons by conditioning feature synthesis on concept activation vectors.

Datasets

  • Pretrained Models: Evaluated primarily on InceptionV1 trained on ImageNet-1k; transferability evaluated on ResNet-18 and MobileNetV2.
  • Concept Tokens: The Describable Textures Dataset (DTD) provided concept sets (e.g., striped, honeycombed, scaly, dots, zig-zagged).
  • Background / Unrelated Sets: Random subsets sampled from ImageNet-1k (e.g., 70 distinct sets of 50 negative images; 1,000 images for the unrelated centroid calculation).

No new public benchmark datasets were introduced.

Key Insights and Ideas

  • Concept Tokens are a Single Point of Failure: Poisoning a small, reusable library of concept images compromises explanations across thousands of unrelated model inferences downstream.
  • Agnostic to Explainability Frameworks: By attacking feature representations directly rather than an interpretability loss, a single perturbed concept set breaks both linear direction metrics (TCAV) and generative priors (FFV).
  • Cross-Layer and Cross-Architecture Transfer: Perturbations optimized for one layer disrupt TCAV evaluations in other layers. Attacks developed on InceptionV1 also transferred to ResNet-18 (early blocks) and MobileNetV2.
  • Stealthy to Qualitative Checks: Inspecting attacked Concept Activation Vectors via DeepDream produces visualizations that look surprisingly normal to human evaluators (with minor color/hue shifts), meaning qualitative spot-checks fail to detect the attack.
  • Inherent Brittleness to Noise: Adding simple Gaussian noise with matching variance also degraded TCAV and FFV scores, revealing that CBIMs may be fragile even under benign distribution shifts.

Key Results

  • TCAV Collapse: For the "striped" concept on the "zebra" class, baseline magnitude scores (~$0.66$ to $0.90$) dropped significantly across layers (falling as low as $0.17 \pm 0.05$ to $0.32 \pm 0.05$ under TP attacks). Similar severe suppressions occurred for the "scaly" concept across four snake classes.
  • FFV Disruption: On 1,760 InceptionV1 channel neurons across four layers, the Fréchet Inception Distance (FID) between clean FFV runs was $0.26$. When concept sets were attacked, FID jumped to $1.34$–$1.39$, quantitatively confirming that the synthesized features lost their intended concept conditioning.
  • Cross-Model Transfer: TP attacks crafted on InceptionV1 caused substantial drops in TCAV magnitude when evaluated on the first block of ResNet-18 and all tested layers of MobileNetV2.

Limitations and Future Work

  • Scope of Modalities and Methods: Experiments were limited to computer vision and two interpretability algorithms (TCAV and FFV). The authors note the need to evaluate NLP architectures and averaging-based visualization approaches (e.g., Activation Atlases).
  • Attack Surface Scope: The attack strictly targets positive concept tokens. Attacking negative examples, model parameters directly, or engineering "triggered" concept backdoors was left for future work.
  • Uneven Architecture Transfer: Transferability was less effective on later residual blocks (Blocks 2 and 4 of ResNet-18).
  • Defense Mechanisms: The paper focuses on vulnerability discovery; designing robust aggregation methods or sanitization defenses for concept sets remains open.

Summary updated

Abstract

Methods for model explainability have become increasingly critical for testing the fairness and soundness of deep learning. A number of explainability techniques have been developed which use a set of examples to represent a human-interpretable concept in a model's activations. In this work we show that these explainability methods can suffer the same vulnerability to adversarial attacks as the models they are meant to analyze. We demonstrate this phenomenon on two well-known concept-based approaches to the explainability of deep learning models: TCAV and faceted feature visualization. We show that by carefully perturbing the examples of the concept that is being investigated, we can radically change the output of the interpretability method, e.g. showing that stripes are not an important factor in identifying images of a zebra. Our work highlights the fact that in safety-critical applications, there is need for security around not only the machine learning pipeline but also the model interpretation process.
BibTeX
@inproceedings{brown2022brittle,
  author = {Davis Brown and Henry Kvinge},
  title = {{Brittle interpretations: The Vulnerability of TCAV and Other Concept-based Explainability Tools to Adversarial Attack}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=a3hQPNqIFk6},
}

Loading related content...

Figure