Interpreting the Second-Order Effects of Neurons in CLIP

Yossi Gandelsman, Alexei A. Efros, Jacob Steinhardt

TL;DR

Interpreting individual neurons in CLIP is notoriously difficult because their direct outputs are negligible and standard ablation techniques are masked by model self-repair. This paper introduces the "second-order lens" to trace how neurons influence the final output by flowing through subsequent attention heads. Using this method, the authors reveal that CLIP neurons are highly sparse, rank-1, and polysemantic, allowing them to automatically mass-produce semantic adversarial images and achieve state-of-the-art zero-shot image segmentation.

What Problem Does This Paper Address?

While vision-language models like CLIP have revolutionized zero-shot classification and image generation, understanding their internal mechanics remains a challenge. Mechanistic interpretability typically relies on two approaches:

  1. Direct effects (Logit Lens): Analyzing the direct flow from a component to the output via the residual stream. In CLIP, individual MLP neurons have a negligible direct effect.
  2. Indirect effects (Ablation): Turning off a neuron to see what breaks. However, neural networks store information redundantly, meaning models often deploy "self-repair" mechanisms where other neurons compensate for the missing one, obscuring its true function.

To bypass these limitations, the authors tackle a fundamental question: How can we faithfully isolate and automatically describe the functional role of individual neurons in a multimodal model?

Methods

The authors introduce the second-order lens, which mathematically isolates a neuron's total contribution to the output by tracing its flow through all subsequent Multi-Head Self-Attention (MSA) layers.

Figure 2: First/Second-order effects. The first order is the flow coming from a neuron to the projection layer and the output (blue). The sec- ond order goes from a single neuron through all the consecutive attention heads, to the projection layer, and to the output (pink).

The framework consists of several key steps:

  • Second-Order Effect Extraction: They compute the additive contribution of a neuron's activation as it interacts with the Output-Value (OV) matrices of later attention heads.
  • Sparse Text Decomposition: They discover that a neuron's second-order effect vector is approximately rank-1 (pointing in a single direction in CLIP's joint space). They use Orthogonal Matching Pursuit (OMP) to decompose this direction into a sparse linear combination of text embeddings from a dictionary (e.g., common English words or automated ImageNet descriptions).
  • Semantic Adversarial Generation: To exploit neuron vulnerabilities, they identify neurons controlling the boundary between two classes (e.g., dog vs. deer). They extract text concepts that spuriously activate these neurons, use an LLM (LLaMA3) to write a natural scene description blending the true class with these spurious cues, and generate adversarial images using a text-to-image model (DeepFloyd IF).
  • Zero-Shot Segmentation: They identify neurons highly aligned with a target class name, then average and binarize their spatial activation maps to generate segmentation masks without any training.

Datasets

The approach was evaluated using the following datasets:

  • ImageNet: The training set was used to compute baseline neuron activation means, and the validation set was used to evaluate classification accuracy drops during ablation.
  • ImageNet-R: Used to verify that the empirical properties of second-order effects generalize to out-of-distribution data.
  • CIFAR-10: Pairs of classes were selected to evaluate the semantic adversarial generation framework.
  • ImageNet-Segmentation: Used to quantitatively benchmark zero-shot segmentation performance against existing explainability baselines.

Key Insights and Ideas

  • Late-Layer Concentration: Significant second-order effects are not evenly distributed; they heavily concentrate in moderately late MLP layers (layers 8–10 in a 12-layer ViT-B-32 model), right before the attention layers that write directly to the output.
  • Extreme Selectivity: Neurons are highly sparse; an individual neuron's second-order effect is only significant for less than 2% of images.
  • Rank-1 Simplicity: Despite the non-linear complexities of Transformers, a single neuron's downstream impact can be cleanly modeled as a single vector direction in CLIP's joint text-image space.
  • Polysemantic Superposition: The text decompositions confirm that individual neurons are polysemantic—responding to multiple, often unrelated concepts (e.g., a single neuron activating for both "yacht" and "cabriolet" because they share latent visual features like sleek luxury profiles).
Figure 5: Images with largest second-order effect norm per neuron. We present the top images from 10% of ImageNet validation set for the neurons in Table 2. Note that neurons are polysemantic - they have large second-order effects on images that show multiple concepts (e.g. cars and boats). See top-50 images in Figure 13.

Key Results

  • Ablation Impact: Simultaneously mean-ablating second-order effects in layer 9 drops ImageNet validation accuracy from ~63% to 29.6%, proving these paths are critical for model performance (whereas ablating indirect effects only drops accuracy to 52.3% due to self-repair).
  • Adversarial Success: The automated semantic adversarial pipeline successfully fools CLIP. For instance, in the dog → deer task, it achieves a 22.7% deception rate, significantly outperforming standard indirect effect or word-similarity baselines.
  • State-of-the-Art Segmentation: The zero-shot segmentation method achieves 59.0% mIoU on ImageNet-Segmentation, outperforming previous explainability methods like TextSpan (58.1%) and Chefer et al. (49.1%).

Limitations and Future Work

  • Attention Value Limitation: The current formulation only tracks how neurons flow through attention values, ignoring their potential impact on attention queries and keys (which alter attention map patterns).
  • Neuron-Neuron Interactions: The method treats neurons independently, ignoring mutual inhibition or amplification effects within the same layer.
  • Pipeline Dependencies: The adversarial generation pipeline relies heavily on the coherence of external LLMs and text-to-image models, introducing external failure modes (e.g., when the text-to-image model fails to render a prompt element accurately).

Summary updated

Abstract

We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual stream to the output) or the indirect effects (overall contribution) fails to capture the neurons' function in CLIP. Therefore, we present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output. We find that these effects are highly selective: for each neuron, the effect is significant for <2% of the images. Moreover, each effect can be approximated by a single direction in the text-image space of CLIP. We describe neurons by decomposing these directions into sparse sets of text representations. The sets reveal polysemantic behavior - each neuron corresponds to multiple, often unrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, we mass-produce "semantic" adversarial examples by generating images with concepts spuriously correlated to the incorrect class. Additionally, we use the second-order effects for zero-shot segmentation, outperforming previous methods. Our results indicate that an automated interpretation of neurons can be used for model deception and for introducing new model capabilities.
BibTeX
@inproceedings{gandelsman2025interpreting,
  author = {Yossi Gandelsman and Alexei A. Efros and Jacob Steinhardt},
  title = {{Interpreting the Second-Order Effects of Neurons in CLIP}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=GPDcvoFGOL},
}
Figure