Data Matters Most: Auditing Social Bias in Contrastive Vision Language Models

Zahraa Al Sahili, Ioannis Patras, Matthew Purver

Abstract

Vision-language models (VLMs) deliver strong zero-shot recognition but frequently inherit social biases from their training data. We systematically disentangle three design factors -- model size, training-data scale, and training-data source -- by comparing CLIP and OpenCLIP, two models that share an identical contrastive objective yet differ in encoder width and in the image-text corpora on which they are pre-trained (400M proprietary pairs vs. 400M/2B LAION). Across balanced face-analysis benchmarks, enlarging the encoder reduces gender skew in CLIP but amplifies both gender and racial skew in OpenCLIP; increasing the LAION corpus from 400M to 2B further increases OpenCLIP bias. At matched model and data budgets, substituting proprietary data with LAION improves gender fairness while increasing racial skew, underscoring data source as the primary driver of bias patterns. We also evaluate three post-hoc, test-time debiasing strategies -- Bias Prompts, Prompt Array, and SANER. Debiasing reduces but does not eliminate harm, and its effectiveness is source- and size-dependent: Bias Prompts most effectively reduce gender skew in CLIP at smaller model sizes, whereas Prompt Array and SANER more reliably reduce racial skew in OpenCLIP; scaling LAION reconfigures which method is most fair. Taken together, these findings challenge the assumption that bigger models or datasets are automatically fairer and foreground training data source as the key determinant of both bias and mitigation efficacy.

Introduction

Contrastive vision-language models like CLIP have revolutionized zero-shot image recognition, but they are notorious for inheriting harmful stereotypes from the massive, uncurated web-scrapes used to train them. Previous research has documented these biases, but it often failed to "disentangle" the causes. Is a model biased because it’s too small to learn nuance? Because the dataset is too small? Or because the data source itself is toxic?

This study addresses this gap by conducting a controlled study that holds the training objective constant while systematically varying three levers: model size (encoder width), training-data scale (400M vs. 2B pairs), and training-data source (OpenAI’s proprietary data vs. the public LAION crawl).

Methodology

The researchers compared OpenAI’s CLIP and the open-source OpenCLIP. Both use the same contrastive loss but differ in their training "ingredients." To measure bias, they used three specific probing tasks:

  1. Denigration (Crime vs. Non-Human): Testing if certain demographics are more likely to be associated with concepts like "thief" or "gorilla."
  2. Communion: Probing interpersonal traits like "trustworthy" vs. "dishonest."
  3. Agency: Probing status-related traits like "powerful" vs. "passive."

Evaluation was performed on FairFace (10,850 balanced portraits) and PATA (Protected Attribute Tag Association), which includes scene context to ensure a robust audit across different visual environments.

Scaling is not a Panacea

A critical finding of this audit is that scaling up model size does not automatically mitigate bias. In CLIP, a larger encoder (moving from B/32 to L/14) generally reduced gender skew. However, in OpenCLIP, the exact same size increase amplified both gender and racial skew. This suggests that parameter count is not a universal "fairness lever"; its effect depends entirely on the data it processes.

Figure 2: Bias Δs by Model Size for gender (left) and race (right). Positive values indicate a reduction in bias when scaling from B/32 to L/14, while negative values indicate an amplification.

Data Source as the Primary Driver

The study highlights that the source of training data is the most critical factor in determining a model's bias profile. At matched budgets, swapping proprietary data for LAION improved gender fairness but significantly worsened racial skew. Furthermore, expanding the LAION dataset from 400M to 2B samples did not "dilute" bias; instead, it often doubled gender skew for smaller models, effectively ossifying the demographic profiles present in the smaller crawl.

Figure 6: Summary of bias score changes (ΔS) across model size, data size, and data decomposition. Red values indicate bias reduction; blue values indicate bias increase.

The "Accuracy-Fairness Paradox" was also observed: the model with the highest zero-shot accuracy (OpenCLIP-L/14 on LAION-2B) also produced the highest rate of criminal misclassification and the largest racial skew.

Mitigation Strategies and Limitations

The authors evaluated three post-hoc debiasing strategies: Bias Prompts, Prompt Array, and SANER. No single strategy was universally effective. Bias Prompts worked best for gender in CLIP, while Prompt Array and SANER were superior for racial skew in OpenCLIP. However, reducing bias on one axis often led to an increase on another—a phenomenon termed "bias transfer."

The study is limited by its focus on binary gender and seven race categories, and the proprietary nature of OpenAI's data prevents a direct audit of its composition. The authors advocate for "data-centric" interventions, suggesting that web crawls should be rebalanced at the source rather than attempting to patch models after the fact.

Conclusion

This systematic audit demonstrates that neither model nor data scaling is a guaranteed path to fairness. Training data source remains the primary determinant of social bias in VLMs. Future evaluations must consider all bias axes at the specific model size intended for deployment to ensure responsible AI development.

Summary updated

Abstract

Vision-language models (VLMs) deliver strong zero-shot recognition but frequently inherit social biases from their training data. We systematically disentangle three design factors -- model size, training-data scale, and training-data source -- by comparing CLIP and OpenCLIP, two models that share an identical contrastive objective yet differ in encoder width and in the image-text corpora on which they are pre-trained (400M proprietary pairs vs. 400M/2B LAION). Across balanced face-analysis benchmarks, enlarging the encoder reduces gender skew in CLIP but amplifies both gender and racial skew in OpenCLIP; increasing the LAION corpus from 400M to 2B further increases OpenCLIP bias. At matched model and data budgets, substituting proprietary data with LAION improves gender fairness while increasing racial skew, underscoring data source as the primary driver of bias patterns. We also evaluate three post-hoc, test-time debiasing strategies -- Bias Prompts, Prompt Array, and SANER. Debiasing reduces but does not eliminate harm, and its effectiveness is source- and size-dependent: Bias Prompts most effectively reduce gender skew in CLIP at smaller model sizes, whereas Prompt Array and SANER more reliably reduce racial skew in OpenCLIP; scaling LAION reconfigures which method is most fair. Taken together, these findings challenge the assumption that bigger models or datasets are automatically fairer and foreground training data source as the key determinant of both bias and mitigation efficacy. We release code and evaluation scripts to enable transparent, reproducible auditing of future VLMs.
BibTeX
@article{sahili2025data,
  author = {Zahraa Al Sahili and Ioannis Patras and Matthew Purver},
  title = {{Data Matters Most: Auditing Social Bias in Contrastive Vision Language Models}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=3vF2fn9owm},
}
Figure