Abstract
Methods for model explainability have become increasingly critical for testing the fairness and soundness of deep learning. Concept-based interpretability techniques, which use a small set of human-interpretable concept exemplars in order to measure the influence of a concept on a model’s internal representation of input, are an important thread in this line of research. In this work we show that these explainability methods can suffer the same vulnerability to adversarial attacks as the models they are meant to analyze. We demonstrate this phenomenon on two well-known concept-based interpretability methods: TCAV and faceted feature visualization. We show that by leveraging the geometry of the problem and carefully perturbing the examples of the concept that is being investigated, we can radically change the output of the interpretability method. The attacks that we propose can either induce positive interpretations (polka dots are an important concept for a model when classifying zebras) or negative interpretations (stripes are not an important factor in identifying images of a zebra). Our work highlights the fact that in safety-critical applications, there is need for security around not only the machine learning pipeline but also the model interpretation process.
Introduction
As deep learning models are increasingly deployed in high-stakes fields like healthcare and finance, the need for "explainability" has become paramount. We need to know why a model made a specific prediction to trust it. Concept-Based Interpretability Methods (CBIMs) have emerged as a popular solution. Instead of looking at individual pixels, CBIMs use a small collection of human-understandable examples (tokens)—like images of stripes or polka dots—to measure how much a "concept" influences a model's internal state.
However, these methods rely on a small, centralized set of tokens. If these tokens are compromised, the entire explanation pipeline breaks. This paper explores whether an adversary can exploit this vulnerability to "fairwash" a biased model or create false justifications for model behavior.
Methodology: The Token Pushing Attack
The authors introduce the Token Pushing (TP) attack. This attack targets the "linear probe" mechanism—a simple linear classifier used by many CBIMs to define a Concept Activation Vector (CAV) in the model's hidden layers. The attack uses Projected Gradient Descent (PGD) to apply nearly invisible perturbations to the concept images.
They propose two versions of the attack:
- Untargeted Attack: Perturbs concept tokens so their hidden representations shift toward a "neutral" centroid of unrelated images. This makes a truly important concept (like "stripes" for a zebra) appear irrelevant to the model.
- Targeted Attack: Perturbs tokens so their hidden representations align with a different target concept. This can trick the tool into reporting that a completely irrelevant concept (like "corgis") is a primary factor in a classification (like "honeycomb").
Experimental Results
The study utilizes standard benchmarks including ImageNet-1k, the Describable Textures Dataset (DTD) for abstract concepts, and the Caltech-UCSD Birds 200 (CUB) for fine-grained classification. The TP attack was tested against two major frameworks: TCAV (Testing with Concept Activation Vectors) and FFV (Faceted Feature Visualization).
Key results include:
- TCAV Manipulation: In an untargeted attack on the 'striped' concept for zebra classification, the TCAV magnitude score dropped from a baseline of ~0.7–0.9 down to ~0.2–0.4, effectively "hiding" the model's reliance on stripes.
- Targeted Success: The authors successfully made 'corgi' and 'dumbbell' images appear as significant concepts for 'honeycomb' classification, despite having no logical connection.
- FFV Distortion: For Faceted Feature Visualization, the TP attack caused the generated visualizations to lose all semantic connection to the original concept. The perceptual difference (measured via FID score) was nearly five times higher than natural variation.
Discussion and Conclusion
The most striking insight is that the tools we use to audit models are just as susceptible to manipulation as the models themselves. Unlike standard adversarial attacks that must perturb every input to change a prediction, a TP attack only needs to perturb a small, centralized set of concept tokens once to affect all subsequent explanations involving that concept. Furthermore, the authors found that attacks developed on one model architecture often successfully "transfer" to others.
Ultimately, this work serves as a call to action for the research community to develop interpretability tools that are "robust by design" against adversarial manipulation, ensuring that model audits remain a reliable source of truth in safety-critical applications.
Summary updated