Introduction
As deep learning models are increasingly deployed in high-stakes fields like healthcare and criminal justice, the need for transparency has led to the development of Concept-Based Interpretability Methods (CBIMs). Tools such as Testing with Concept Activation Vectors (TCAV) and Faceted Feature Visualization (FFV) aim to provide human-understandable explanations by mapping a model's internal representations to high-level concepts (e.g., "stripes" or "bubbly"). However, this research demonstrates that these tools possess a critical security gap: they are highly vulnerable to adversarial attacks that can manipulate explanations without altering the model's actual predictions.
Problem Statement: The Fragility of Concept Tokens
CBIMs typically rely on a small set of "concept tokens"—a collection of images representing a specific attribute—to define a concept direction in the model's feature space. This reliance on a limited, centralized dataset creates a "single point of failure." An adversary who can subtly perturb these tokens can "gaslight" users, making irrelevant concepts appear essential or hiding the importance of critical features. Because these perturbations are designed to be imperceptible to humans, the manipulated explanations can lead to a false sense of trust in a model's decision-making process.
Methodology: The Token Pushing (TP) Attack
The authors introduce the Token Pushing (TP) attack, which targets the linear probe mechanism common to many CBIMs. A linear probe is a simple classifier trained on the internal embeddings of concept tokens to identify a Concept Activation Vector (CAV).
The TP attack utilizes Projected Gradient Descent (PGD) to add adversarial noise to the concept tokens. The objective is to shift the tokens' embeddings in a specific hidden layer:
- Untargeted Attack: Pushes tokens toward the centroid of random, unrelated images, causing the concept to appear irrelevant.
- Targeted Attack: Pushes tokens toward the centroid of a different, specific concept. For instance, "corgi" tokens can be shifted to appear like "honeycomb" textures to the model's internal layers.
Crucially, the attack is often transferable across different model architectures, suggesting that diverse models learn similar internal representations for high-level concepts.
Experimental Setup
The researchers evaluated the TP attack using several standard benchmarks:
- ImageNet-1k: Primary classification models included InceptionV1, ResNet-18, MobileNetV2, and Vision Transformers (ViT).
- Describable Textures Dataset (DTD): Source for abstract concepts like "striped" and "bubbly."
- Caltech-UCSD Birds 200 (CUB): Used for fine-grained attribute concepts such as "spatulate bill shape."
Key Results
TCAV Manipulation and Targeted Deception
The effectiveness of the TP attack was demonstrated through significant shifts in TCAV magnitude scores. In untargeted attacks on a "zebra" classifier, the importance of the "striped" concept dropped from a high baseline (~0.8) to near zero.
More strikingly, targeted attacks successfully made unrelated concepts appear vital. As shown in the experimental results, concepts like "corgis," "dumbbells," and "bubbly textures" were manipulated to appear highly important for the "honeycomb" class, despite having no logical connection to the target category.
FFV Distortion and Visualization Failure
The attack also severely impacts Faceted Feature Visualization (FFV). By shifting the concept tokens, the attack increased the perceptual distance (measured via FID score) between original and attacked visualizations by a factor of five. Standard sanity checks, such as using DeepDream to visualize the attacked concepts, failed to detect the corruption, as the resulting images still appeared qualitatively similar to the originals to human observers.
Discussion and Limitations
The findings highlight that the robustness of an explanation is as vital as the robustness of the model. If interpretability tools can be easily manipulated, they cannot serve as reliable safety checks in sensitive applications.
However, the study has limitations. It focused exclusively on image classification; the vulnerability of CBIMs in Natural Language Processing (NLP) remains an open question. Additionally, the authors primarily targeted tools that rely on small, static sets of concept tokens. Future research should explore whether methods that aggregate over larger datasets offer better resistance to such adversarial manipulation.
Conclusion
This work exposes a significant vulnerability in current concept-based interpretability methods. By demonstrating that explanations can be decoupled from model behavior through nearly invisible perturbations, the authors call for a more rigorous approach to evaluating the security and reliability of AI transparency tools.
Summary updated