TL;DR
Concept-based explainability tools like TCAV and Faceted Feature Visualization (FFV) help humans understand deep networks by mapping internal activations to user-provided concept examples ("tokens"). This paper introduces Token Pushing (TP) attacks, demonstrating that applying imperceptible perturbations to these centralized concept tokens can manipulate the explanations—fooling TCAV into reporting that stripes are irrelevant for identifying zebras, and breaking neuron visualizations without changing the underlying model.
What Problem Does This Paper Address?
Deep neural networks are often deployed in safety-critical domains like healthcare and finance, where interpretability is essential for auditing decisions. Concept-Based Interpretability Methods (CBIMs) interrogate models using collections of positive examples (concept tokens) representing human-understandable concepts (e.g., "stripes," "scales," or "textures").
While adversarial attacks on model predictions and saliency maps are well-documented, the security of CBIMs has remained largely unexamined. Because concept datasets are often curated centrally and reused across multiple audit pipelines, they introduce a single point of failure: what happens if an adversary tampers with the concept tokens themselves?
Methods
The authors formulate a formal threat model for post-hoc CBIMs and propose the Token Pushing (TP) attack:
- Threat Model: The attacker has read/write access to the positive concept tokens ($P_C$) and read access to at least a surrogate model, but does not need access to the target dataset/neuron, nor do they need to know which specific interpretability algorithm will be run.
- Objective Formulation: Instead of optimizing against a specific explainability metric, the attack disrupts the underlying geometric assumption of CBIMs—that semantically related inputs cluster closely in intermediate activation space.
- Centroid PGD Attack: For a target layer $\ell$, the attacker calculates the centroid $\mu_T$ of representations of an unrelated image set $U_C$. Then, using Projected Gradient Descent (PGD with $\ell_\infty$ budget $\epsilon = 4/255$ over 20 iterations), each positive concept token $x_i^P \in P_C$ is perturbed to minimize its distance to $\mu_T$: $$\delta_i := \arg\min_{\|\delta\|_\infty \le \epsilon} \|f_\ell(x_i^P + \delta) - \mu_T\|$$
- Target Explainability Methods:
- TCAV (Testing with Concept Activation Vectors): Tests whether a concept vector aligns with the directional derivative of model predictions.
- Faceted Feature Visualization (FFV): Disambiguates polysemantic neurons by conditioning feature synthesis on concept activation vectors.
Datasets
- Pretrained Models: Evaluated primarily on InceptionV1 trained on ImageNet-1k; transferability evaluated on ResNet-18 and MobileNetV2.
- Concept Tokens: The Describable Textures Dataset (DTD) provided concept sets (e.g., striped, honeycombed, scaly, dots, zig-zagged).
- Background / Unrelated Sets: Random subsets sampled from ImageNet-1k (e.g., 70 distinct sets of 50 negative images; 1,000 images for the unrelated centroid calculation).
No new public benchmark datasets were introduced.
Key Insights and Ideas
- Concept Tokens are a Single Point of Failure: Poisoning a small, reusable library of concept images compromises explanations across thousands of unrelated model inferences downstream.
- Agnostic to Explainability Frameworks: By attacking feature representations directly rather than an interpretability loss, a single perturbed concept set breaks both linear direction metrics (TCAV) and generative priors (FFV).
- Cross-Layer and Cross-Architecture Transfer: Perturbations optimized for one layer disrupt TCAV evaluations in other layers. Attacks developed on InceptionV1 also transferred to ResNet-18 (early blocks) and MobileNetV2.
- Stealthy to Qualitative Checks: Inspecting attacked Concept Activation Vectors via DeepDream produces visualizations that look surprisingly normal to human evaluators (with minor color/hue shifts), meaning qualitative spot-checks fail to detect the attack.
- Inherent Brittleness to Noise: Adding simple Gaussian noise with matching variance also degraded TCAV and FFV scores, revealing that CBIMs may be fragile even under benign distribution shifts.
Key Results
- TCAV Collapse: For the "striped" concept on the "zebra" class, baseline magnitude scores (~$0.66$ to $0.90$) dropped significantly across layers (falling as low as $0.17 \pm 0.05$ to $0.32 \pm 0.05$ under TP attacks). Similar severe suppressions occurred for the "scaly" concept across four snake classes.
- FFV Disruption: On 1,760 InceptionV1 channel neurons across four layers, the Fréchet Inception Distance (FID) between clean FFV runs was $0.26$. When concept sets were attacked, FID jumped to $1.34$–$1.39$, quantitatively confirming that the synthesized features lost their intended concept conditioning.
- Cross-Model Transfer: TP attacks crafted on InceptionV1 caused substantial drops in TCAV magnitude when evaluated on the first block of ResNet-18 and all tested layers of MobileNetV2.
Limitations and Future Work
- Scope of Modalities and Methods: Experiments were limited to computer vision and two interpretability algorithms (TCAV and FFV). The authors note the need to evaluate NLP architectures and averaging-based visualization approaches (e.g., Activation Atlases).
- Attack Surface Scope: The attack strictly targets positive concept tokens. Attacking negative examples, model parameters directly, or engineering "triggered" concept backdoors was left for future work.
- Uneven Architecture Transfer: Transferability was less effective on later residual blocks (Blocks 2 and 4 of ResNet-18).
- Defense Mechanisms: The paper focuses on vulnerability discovery; designing robust aggregation methods or sanitization defenses for concept sets remains open.
Summary updated