Interpreting the Second-Order Effects of Neurons in CLIP

Yossi Gandelsman, Alexei A. Efros, Jacob Steinhardt

Abstract

We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual stream to the output) or the indirect effects (overall contribution) fails to capture the neurons' function in CLIP. Therefore, we present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output. We find that these effects are highly selective: for each neuron, the effect is significant for <2% of the images. Moreover, each effect can be approximated by a single direction in the text-image space of CLIP. We describe neurons by decomposing these directions into sparse sets of text representations. The sets reveal polysemantic behavior - each neuron corresponds to multiple, often unrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, we mass-produce "semantic" adversarial examples by generating images with concepts spuriously correlated to the incorrect class. Additionally, we use the second-order effects for zero-shot segmentation and attribute discovery in images. Our results indicate that a scalable understanding of neurons can be used for model deception and for introducing new model capabilities.

Introduction

Interpretability in large-scale vision-language models like CLIP is a significant challenge. While attention heads have been successfully decoded, individual neurons—the fundamental units of Multi-Layer Perceptron (MLP) layers—remain largely opaque. Previous attempts to understand neurons through direct effects (the "logit lens") or ablation studies often fall short due to negligible individual contributions and the model's ability to "self-repair" by shifting computation to redundant neurons.

A new approach presented at ICLR 2025 introduces the second-order lens. This method shifts focus from a neuron's immediate output to its downstream influence as it propagates through subsequent attention heads to reach the final classification.

The Second-Order Lens

The core innovation lies in deriving the "second-order effect" of a neuron. Mathematically, this involves summing a neuron's contributions as they are weighted by all following Multi-Head Self-Attention (MSA) layers. This captures the indirect but significant path the signal takes to influence the final logit.

To translate these high-dimensional vectors into human-interpretable concepts, the researchers employ Sparse Decomposition. By treating a neuron's effect as a vector in CLIP’s joint text-image space, they use Orthogonal Matching Pursuit (OMP) to decompose it into a sparse set of descriptive words (e.g., "dog," "snowy," "yacht"). This allows for the automatic labeling of neurons with specific semantic concepts.

The second-order lens framework (top) and its application in generating adversarial examples via text-based sparse neuron decomposition (bottom).

Key Insights: Polysemanticity and Rank-1 Approximation

The second-order lens reveals several fundamental properties of CLIP's internal representations:

  1. Polysemantic Neurons: Contrary to the "one neuron, one concept" ideal, individual neurons are often polysemantic. A single neuron might activate for multiple, seemingly unrelated concepts, such as "ships" and "cars." This suggests that CLIP utilizes its high-dimensional space to pack diverse information efficiently.
  2. Rank-1 Approximation: Despite the network's depth, a neuron’s second-order effect can be accurately modeled by a single direction in the embedding space. Replacing the full effect with this one-dimensional approximation results in negligible loss in model accuracy, simplifying the interpretability task.
  3. Sparsity and Selectivity: Neurons are highly selective; any given neuron typically contributes significantly to fewer than 2% of images. These impactful effects are most concentrated in the later layers of the Vision Transformer (layers 8-10).
Visual evidence of polysemantic neurons. For instance, Neuron #2914 activates for various types of boats and vehicle interiors, demonstrating multi-concept encoding.

Applications and Results

The utility of the second-order lens was demonstrated through two primary applications:

Zero-Shot Segmentation

By identifying neurons associated with specific classes and mapping their spatial activations, the authors developed a zero-shot segmenter. This approach achieved a 59.0 mIoU on ImageNet-Segmentation, surpassing the previous state-of-the-art method, TextSpan (58.1).

Semantic Adversarial Examples

The discovery of polysemantic neurons allowed for the creation of "semantic" adversarial attacks. By identifying neurons that respond to both a target class and a spurious concept, the researchers could fool CLIP without traditional pixel-level perturbations. For example, to misclassify a "dog" as a "deer," they generated images of dogs in scenic forests—a concept identified as spuriously linked to "deer" in specific neurons. This method proved significantly more effective than baseline attacks.

Limitations and Future Outlook

While the second-order lens provides a powerful tool for transparency, it currently focuses on how neurons influence attention values, ignoring potential impacts on attention patterns (keys and queries). Additionally, it treats neurons as independent units, leaving the study of lateral neuron-neuron interactions for future research.

Ultimately, these findings suggest that a scalable understanding of individual neurons can not only reveal model vulnerabilities but also unlock new capabilities, such as surgical fine-tuning to remove unwanted spurious correlations.

Summary updated

Abstract

We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual stream to the output) or the indirect effects (overall contribution) fails to capture the neurons' function in CLIP. Therefore, we present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output. We find that these effects are highly selective: for each neuron, the effect is significant for <2% of the images. Moreover, each effect can be approximated by a single direction in the text-image space of CLIP. We describe neurons by decomposing these directions into sparse sets of text representations. The sets reveal polysemantic behavior - each neuron corresponds to multiple, often unrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, we mass-produce "semantic" adversarial examples by generating images with concepts spuriously correlated to the incorrect class. Additionally, we use the second-order effects for zero-shot segmentation and attribute discovery in images. Our results indicate that a scalable understanding of neurons can be used for model deception and for introducing new model capabilities.
BibTeX
@inproceedings{gandelsman2025interpreting,
  author = {Yossi Gandelsman and Alexei A. Efros and Jacob Steinhardt},
  title = {{Interpreting the Second-Order Effects of Neurons in CLIP}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=GPDcvoFGOL},
}
Figure