Brittle interpretations: The Vulnerability of TCAV and Other Concept-based Explainability Tools to Adversarial Attack

Davis Brown, Henry Kvinge

Introduction

As deep learning models are increasingly deployed in high-stakes fields like healthcare and criminal justice, the need for transparency has led to the development of Concept-Based Interpretability Methods (CBIMs). Tools such as Testing with Concept Activation Vectors (TCAV) and Faceted Feature Visualization (FFV) aim to provide human-understandable explanations by mapping a model's internal representations to high-level concepts (e.g., "stripes" or "bubbly"). However, this research demonstrates that these tools possess a critical security gap: they are highly vulnerable to adversarial attacks that can manipulate explanations without altering the model's actual predictions.

Problem Statement: The Fragility of Concept Tokens

CBIMs typically rely on a small set of "concept tokens"—a collection of images representing a specific attribute—to define a concept direction in the model's feature space. This reliance on a limited, centralized dataset creates a "single point of failure." An adversary who can subtly perturb these tokens can "gaslight" users, making irrelevant concepts appear essential or hiding the importance of critical features. Because these perturbations are designed to be imperceptible to humans, the manipulated explanations can lead to a false sense of trust in a model's decision-making process.

Methodology: The Token Pushing (TP) Attack

The authors introduce the Token Pushing (TP) attack, which targets the linear probe mechanism common to many CBIMs. A linear probe is a simple classifier trained on the internal embeddings of concept tokens to identify a Concept Activation Vector (CAV).

Overview of the Token Pushing attack mechanism. The attack shifts the embeddings of positive tokens ($P_C$) toward a target region, effectively rotating the decision boundary (CAV) of the linear probe.

The TP attack utilizes Projected Gradient Descent (PGD) to add adversarial noise to the concept tokens. The objective is to shift the tokens' embeddings in a specific hidden layer:

  • Untargeted Attack: Pushes tokens toward the centroid of random, unrelated images, causing the concept to appear irrelevant.
  • Targeted Attack: Pushes tokens toward the centroid of a different, specific concept. For instance, "corgi" tokens can be shifted to appear like "honeycomb" textures to the model's internal layers.

Crucially, the attack is often transferable across different model architectures, suggesting that diverse models learn similar internal representations for high-level concepts.

Experimental Setup

The researchers evaluated the TP attack using several standard benchmarks:

  • ImageNet-1k: Primary classification models included InceptionV1, ResNet-18, MobileNetV2, and Vision Transformers (ViT).
  • Describable Textures Dataset (DTD): Source for abstract concepts like "striped" and "bubbly."
  • Caltech-UCSD Birds 200 (CUB): Used for fine-grained attribute concepts such as "spatulate bill shape."

Key Results

TCAV Manipulation and Targeted Deception

The effectiveness of the TP attack was demonstrated through significant shifts in TCAV magnitude scores. In untargeted attacks on a "zebra" classifier, the importance of the "striped" concept dropped from a high baseline (~0.8) to near zero.

More strikingly, targeted attacks successfully made unrelated concepts appear vital. As shown in the experimental results, concepts like "corgis," "dumbbells," and "bubbly textures" were manipulated to appear highly important for the "honeycomb" class, despite having no logical connection to the target category.

TCAV magnitude scores for targeted attacks. The attack successfully inflates the importance of unrelated concepts (corgi, dumbbell, bubbly) for a target class across various network layers.

FFV Distortion and Visualization Failure

The attack also severely impacts Faceted Feature Visualization (FFV). By shifting the concept tokens, the attack increased the perceptual distance (measured via FID score) between original and attacked visualizations by a factor of five. Standard sanity checks, such as using DeepDream to visualize the attacked concepts, failed to detect the corruption, as the resulting images still appeared qualitatively similar to the originals to human observers.

Discussion and Limitations

The findings highlight that the robustness of an explanation is as vital as the robustness of the model. If interpretability tools can be easily manipulated, they cannot serve as reliable safety checks in sensitive applications.

However, the study has limitations. It focused exclusively on image classification; the vulnerability of CBIMs in Natural Language Processing (NLP) remains an open question. Additionally, the authors primarily targeted tools that rely on small, static sets of concept tokens. Future research should explore whether methods that aggregate over larger datasets offer better resistance to such adversarial manipulation.

Conclusion

This work exposes a significant vulnerability in current concept-based interpretability methods. By demonstrating that explanations can be decoupled from model behavior through nearly invisible perturbations, the authors call for a more rigorous approach to evaluating the security and reliability of AI transparency tools.

Summary updated

Abstract

Methods for model explainability have become increasingly critical for testing the fairness and soundness of deep learning. Concept-based interpretability techniques, which use a small set of human-interpretable concept exemplars in order to measure the influence of a concept on a model's internal representation of input, are an important thread in this line of research. In this work we show that these explainability methods can suffer the same vulnerability to adversarial attacks as the models they are meant to analyze. We demonstrate this phenomenon on two well-known concept-based interpretability methods: TCAV and faceted feature visualization. We show that by carefully perturbing the examples of the concept that is being investigated, we can radically change the output of the interpretability method. The attacks that we propose can either induce positive interpretations (polka dots are an important concept for a model when classifying zebras) or negative interpretations (stripes are not an important factor in identifying images of a zebra). Our work highlights the fact that in safety-critical applications, there is need for security around not only the machine learning pipeline but also the model interpretation process.
BibTeX
@inproceedings{brown2022brittle,
  author = {Davis Brown and Henry Kvinge},
  title = {{Brittle interpretations: The Vulnerability of TCAV and Other Concept-based Explainability Tools to Adversarial Attack}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=a3hQPNqIFk6},
}
Figure