Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Models

Association for Computational Linguistics, Ioannis Patras, Matthew Purver, Zahraa Al SahiliUnderline 2025 · DOI · OpenAlex

Abstract

Multilingual vision–language models (VLMs) promise universal image–text retrieval, yet their social biases remain under‑explored. We perform the first systematic audit of four public multilingual CLIP variants—M‑CLIP, NLLB‑CLIP, CAPIVARA‑CLIP, and the debiased SigLIP‑2—covering ten languages that differ in resource availability and morphological gender marking. Contrary to the intuition that multilinguality mitigates bias, every model exhibits stronger gender skew than its English‑only baseline. CAPIVARA‑CLIP shows its largest biases precisely in the low‑resource languages it targets, while the shared encoder of NLLB‑CLIP and SigLIP‑2 transfers English gender stereotypes into gender‑neutral languages. Although SigLIP‑2 reduces agency and communion skews, it inherits—and in caption‑sparse contexts (e.g., Xhosa) amplifies—the English anchor’s crime associations. Aggregated metrics thus mask language‑specific “hot spots,” underscoring the need for fine‑grained, language‑aware bias evaluation in future multilingual VLM research.

Introduction

The development of multilingual Contrastive Language-Image Pre-training (CLIP) models has been a cornerstone in making AI accessible globally. By aligning images with text in hundreds of languages, these models enable cross-lingual search and generation. However, this expansion often relies on an "English anchor"—using English as the central hub for alignment. This study investigates whether this linguistic expansion inadvertently exports and amplifies social biases, particularly regarding race and gender.

Methodology

The researchers audited four prominent multilingual CLIP variants, each representing a different architectural approach:

  • M-CLIP: Uses distillation from an English CLIP teacher to a multilingual text encoder.
  • NLLB-CLIP: Integrates Meta’s "No Language Left Behind" text encoder.
  • CAPIVARA-CLIP: Employs parameter-efficient fine-tuning (LoRA) to adapt the model for specific languages.
  • SigLIP-2: A state-of-the-art model trained on 40 billion pairs with explicit bias-filtering intended to reduce social skews.

The audit spanned ten languages, categorized by resource availability (High vs. Low) and grammatical gender (Gendered vs. Neutral). Bias was measured using zero-shot probing on the FairFace and PATA datasets, focusing on skews in agency, communion, and criminality associations.

The "English Anchor" and Bias Amplification

A primary finding is that multilinguality does not dilute bias; it often acts as a megaphone. Every multilingual model tested showed higher gender skew than its English-only counterpart. This suggests that the process of mapping diverse languages into an English-centric embedding space forces them to inherit Western stereotypical associations.

Low-Resource Vulnerability

In low-resource settings, where training data is sparse and noisy, models rely more heavily on biased "priors." For instance, while SigLIP-2 was designed to be "debiased" in English, it exhibits extreme spikes in racial bias for languages like Xhosa.

Figure 1: Race skew values in low-resource languages, showing the dramatic amplification of crime associations in Xhosa for the SigLIP-2 model.

As seen in Figure 1, the race_crime skew for xh-SigLIP-2 is an order of magnitude higher than in other languages, illustrating how "debiasing" in English does not necessarily translate to fairness in low-resource contexts where caption data is sparse.

Typological Interference: Grammar as a Catalyst

The study also highlights how the grammatical structure of a language influences bias. Highly gendered languages like Spanish and French consistently showed higher bias scores than gender-neutral ones. Interestingly, shared encoders (like those in NLLB-CLIP) were found to "leak" gender stereotypes from English into gender-neutral languages like Turkish and Finnish.

Figure 6: Gender skew values in high-resource languages (English, Spanish, French). Note the high agency skew for CAPIVARA-CLIP in French and the persistent crime associations across models.

Figure 6 demonstrates that even in high-resource languages, models like CAPIVARA-CLIP exhibit significant skews in agency and criminality. While SigLIP-2 successfully reduces agency and communion skews (the yellow and orange bars), it fails to address—and in some cases inherits—the red "crime" associations from its English anchor.

Conclusion

The promise of universal VLMs is currently hampered by a failure to account for language-specific bias dynamics. Aggregated metrics often hide these "hot spots," such as the extreme criminality associations in Xhosa or the agency skews in French. Future research must move away from English-centric alignment and adopt fine-grained, language-aware evaluation and mitigation strategies to ensure that breaking language barriers doesn't mean reinforcing social ones.

Summary updated

Abstract

Multilingual vision--language models (VLMs) promise universal image--text retrieval, yet their social biases remain under‑explored. We perform the first systematic audit of four public multilingual CLIP variants—M‑CLIP, NLLB‑CLIP, CAPIVARA‑CLIP, and the debiased SigLIP‑2—covering ten languages that differ in resource availability and morphological gender marking. Using balanced subsets of \textsc{FairFace} and the \textsc{PATA} stereotype suite in a zero‑shot setting, we quantify race and gender bias and measure stereotype amplification. Contrary to the intuition that multilinguality mitigates bias, \emph{every} model exhibits stronger gender skew than its English‑only baseline. CAPIVARA‑CLIP shows its largest biases precisely in the low‑resource languages it targets, while the shared encoder of NLLB‑CLIP and SigLIP‑2 transfers English gender stereotypes into gender‑neutral languages; loosely coupled encoders largely avoid this leakage. Although SigLIP‑2 reduces agency and communion skews, it inherits—and in caption‑sparse contexts (e.g., Xhosa) amplifies—the English anchor’s crime associations. Highly gendered languages consistently magnify all bias types, yet gender‑neutral languages remain vulnerable whenever cross‑lingual weight sharing imports foreign stereotypes. Aggregated metrics thus mask language‑specific “hot spots,” underscoring the need for fine‑grained, language‑aware bias evaluation in future multilingual VLM research.
BibTeX
@inproceedings{linguistics2025breaking,
  author = {Association for Computational Linguistics and Ioannis Patras and Matthew Purver and Zahraa Al Sahili and Matthew Purver and Zahraa Al Sahili},
  title = {{Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Models}},
  booktitle = {Underline},
  year = {2025},
  doi = {10.48448/ek7s-sm13},
}

Loading related content...

Figure