Pruning the Paradox: How CLIP's Most Informative Heads Enhance Performance While Amplifying Bias

Avinash Madasu, Vasudev Lal, Phillip Howard

Abstract

CLIP is one of the most popular foundation models and is heavily used for many vision-language tasks, yet little is known about its inner workings. As CLIP is increasingly deployed in real-world applications, it is becoming even more critical to understand its limitations and embedded social biases to mitigate potentially harmful downstream consequences. However, the question of what internal mechanisms drive both the impressive capabilities as well as problematic shortcomings of CLIP has largely remained unanswered. To bridge this gap, we study the conceptual consistency of text descriptions for attention heads in CLIP-like models. Specifically, we propose Concept Consistency Score (CCS), a novel interpretability metric that measures how consistently individual attention heads in CLIP models align with specific concepts. Our soft-pruning experiments reveal that high CCS heads are critical for preserving model performance, as pruning them leads to a significantly larger performance drop than pruning random or low CCS heads. Notably, we find that high CCS heads capture essential concepts and play a key role in out-of-domain detection, concept-specific reasoning, and video-language understanding. Moreover, we prove that high CCS heads learn spurious correlations which amplify social biases. These results position CCS as a powerful interpretability metric exposing the paradox of performance and social biases in CLIP models.

Introduction

Contrastive Language-Image Pre-training (CLIP) models serve as the foundation for a wide array of modern AI applications, from zero-shot image classification to text-to-image generation. Despite their ubiquity, these models remain largely "black boxes." While their high performance is well-documented, so too are their tendencies to exhibit significant social biases. This research addresses a fundamental question: which internal mechanisms drive CLIP's capabilities, and are these same mechanisms responsible for its shortcomings? By identifying specific attention heads that encode semantic concepts, the authors explore whether the model's "intelligence" is inextricably linked to its learned spurious correlations.

Methodology: The Concept Consistency Score (CCS)

To map the internal conceptual landscape of CLIP, the researchers developed a multi-stage interpretability framework centered on a new metric: the Concept Consistency Score (CCS).

The CCS Pipeline: From raw attention head outputs to a validated conceptual consistency score.

The process involves several key steps:

  1. Representation Extraction: Utilizing the TEXTSPAN algorithm to generate descriptive text spans for attention heads in the final layers of the image encoder.
  2. Concept Labeling: Employing an LLM to synthesize these descriptions into a single "Concept Label" (e.g., "Shapes," "Locations," "People").
  3. Consistency Scoring: Three independent LLMs evaluate how well the original text descriptions match the assigned label. A head that consistently aligns with its concept receives a high CCS.
  4. Soft-Pruning: To validate the importance of these heads, the authors zero out the attention weights of specific heads and observe the impact on downstream tasks without retraining.

Key Results and the Performance-Bias Paradox

The study's most striking finding is the "performance-bias paradox." High CCS heads—those that most consistently represent specific semantic concepts—are the primary drivers of model accuracy. However, these same heads are also the primary repositories of social bias.

Impact of pruning specific concept-aligned heads (e.g., Location heads for a location task) leads to a much sharper performance decline than pruning unrelated concept heads.

In experiments on the Country-211 dataset, pruning "Location" heads in the ViT-B-16-OpenAI model caused accuracy to drop from 22.81% to 14.09%, whereas pruning "Animals" heads had negligible impact. This demonstrates that these heads are not redundant but are specialized experts for specific domains.

Crucially, social biases (like gender or racial stereotypes) are concentrated in these same high-performing heads. For instance, in the SocialCounterFactuals dataset, pruning High CCS heads in the ViT-B-32-OpenAI model significantly reduced "Race MaxSkew" from 4.1 to 1.2. This suggests that CLIP learns concepts and their associated stereotypes using the same specialized internal structures.

Insights and Implications

  • Specialization vs. Redundancy: High CCS heads are the model's workhorses. Pruning them causes a massive performance collapse, while pruning Low CCS heads (which lack clear semantic focus) has minimal effect.
  • Architecture and Robustness: Larger models like ViT-L-14 exhibit more distributed knowledge, making them more resilient to pruning than smaller models like ViT-B-32, which rely on a few highly specialized heads.
  • Zero-Shot Mitigation: The ability to identify high-bias heads via CCS allows for bias reduction without expensive fine-tuning, though it often involves a "tax" on overall accuracy.

Limitations

While CCS provides a powerful lens into CLIP, it relies on the judgment of other LLMs for labeling and scoring. Although validated against human annotations, this creates a model-to-model dependency. Furthermore, the current study focuses on the final layers of the image encoder; future research is needed to determine if these patterns hold in earlier layers or within the text encoder.

Summary updated

Abstract

CLIP is one of the most popular foundation models and is heavily used for many vision-language tasks, yet little is known about its inner workings. As CLIP is increasingly deployed in real-world applications, it is becoming even more critical to understand its limitations and embedded social biases to mitigate potentially harmful downstream consequences. However, the question of what internal mechanisms drive both the impressive capabilities as well as problematic shortcomings of CLIP has largely remained unanswered. To bridge this gap, we study the conceptual consistency of text descriptions for attention heads in CLIP-like models. Specifically, we propose Concept Consistency Score (CCS), a novel interpretability metric that measures how consistently individual attention heads in CLIP models align with specific concepts. Our soft-pruning experiments reveal that high CCS heads are critical for preserving model performance, as pruning them leads to a significantly larger performance drop than pruning random or low CCS heads. Notably, we find that high CCS heads capture essential concepts and play a key role in out-of-domain detection, concept-specific reasoning, and video-language understanding. Moreover, we prove that high CCS heads learn spurious correlations which amplify social biases. These results position CCS as a powerful interpretability metric exposing the paradox of performance and social biases in CLIP models.
BibTeX
@misc{madasu2025pruning,
  author = {Avinash Madasu and Vasudev Lal and Phillip Howard},
  title = {{Pruning the Paradox: How CLIP's Most Informative Heads Enhance Performance While Amplifying Bias}},
  year = {2025},
  eprint = {2503.11103},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.11103},
}
Figure