When vision-language models (VLMs) like CLIP are trained on billions of uncurated image-text pairs from the internet, they learn more than just the literal mapping between words and pixels. They internalize the structural inequalities, cultural myths, and prejudices embedded within that data. As Hoin Jung et al. describe in A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks, these models act as "mirrors of prejudice," mapping neutral concepts like "doctor" or "nurse" disproportionately to specific genders or races.
Understanding why this happens requires looking at how multimodal representations are formed, how the interaction between text and vision can actually worsen bias compared to text alone, and how researchers are beginning to surgically untangle these associations.
The Multimodal Feedback Loop
Why does combining text and vision worsen bias? Intuitively, one might think that adding a visual modality provides grounding that anchors text to objective reality. However, Yi Zhang et al. demonstrated in Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models that multimodal pre-training often yields higher levels of social bias than unimodal language modeling.
The reason lies in the contrastive objective. When a model forces text embeddings and image embeddings into a shared latent space, it exploits any statistical shortcut available to maximize alignment. If the training data predominantly features men in engineering roles and women in caretaking roles, the model doesn't just learn the definitions of "engineer" and "nurse"—it binds the semantic concept of the profession directly to demographic features like gender or race.
This leads to what Ali Abdollahi et al. term "gender-activity binding bias" in GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Models. When presented with complex scenes containing multiple people, VLMs struggle with fine-grained visual reasoning and instead default to stereotypical heuristics, incorrectly assigning activities based on the gender of the subjects. Similarly, Md Mehrab Tanjim et al. found in Discovering and Mitigating Biases in CLIP-based Image Editing that models frequently treat demographic features as proxies for objects or professions, prioritizing a person's appearance over relevant contextual cues like tools or environments.
From Objective Descriptions to Subjective Leaks
Bias in VLMs is not always explicitly visible; it often remains latent until triggered by specific types of interaction. In BiasDora: Exploring Hidden Biased Associations in Vision-Language Models, Chahat Raj et al. showed that the transition from objective factual reporting to subjective interpretation acts as a major catalyst for bias leakage. While a VLM might describe an image neutrally when asked for a literal caption, prompting the model for an evaluative judgment or a personality trait causes deep-seated stereotypical associations to surface.
This manifests in harmful ways across multiple dimensions:
- Racialized Personality Stereotypes: Vatsal Baherwani et al. in Racial and Gender Stereotypes Encoded Into CLIP Representations revealed that CLIP encodes racialized personality stereotypes, assigning higher confidence scores for traits like diligence or intelligence to specific ethnic groups based on cultural myths.
- Asymmetric Sexualization: Aylin Caliskan in Artificial Intelligence, Bias, and Ethics highlighted a stark disparity in how genders are portrayed, with models frequently associating female subjects with sexualized contexts while keeping male subjects in professional or neutral settings.
Probing the Latent Space with Counterfactuals
To diagnose and quantify these biases accurately, researchers cannot rely on random web images, as real-world confounding variables make it difficult to isolate the exact cause of a model's skewed prediction. Instead, the field has converged on using counterfactual evaluation.
The intuition behind a counterfactual probe is simple: take an image-text pair, alter exactly one demographic attribute (such as race, gender, or body type) while keeping every other pixel and contextual element identical, and observe how the model's output changes.
Original Image: [Man in scrubs in a hospital] ──> Model Prediction: "A doctor"
Counterfactual: [Woman in scrubs in a hospital] ──> Model Prediction: "A nurse"
Phillip Howard et al. utilized this approach at scale in Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals, generating millions of responses to synthetic counterfactual images. Their findings revealed severe biases against specific physical characteristics, such as body type, which models systematically associated with lower professional competency. By isolating the demographic variable, counterfactuals provide an unconfounded metric of how heavily a model relies on superficial identity traits rather than actual semantic content.
Surgically Untangling Bias from Knowledge
Once biases are identified and measured, the challenge shifts to mitigation. How do we remove these harmful associations without destroying the model's core zero-shot capabilities and general intelligence?
A naive approach might involve retraining the model from scratch on heavily curated data, but this is computationally prohibitive. Instead, recent work focuses on parameter-efficient adjustments.
One effective strategy is parameter-efficient fine-tuning. Leander Girrbach et al. demonstrated in Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs) that using Low-Rank Adaptation (LoRA) offers an optimal balance, successfully reducing occupational bias while preserving the model's downstream intelligence.
Another angle targets the embedding space directly. In Joint Vision-Language Social Bias Removal for CLIP, Haoyu Zhang et al. proposed aligning the disparate bias distributions found within the separate image and text modalities before applying counterfactual debiasing.
Alternatively, Amirabbas Afzali et al. adapted Preference Optimization (PO) techniques from large language models to vision-language representations in Aligning Visual Contrastive learning models via Preference Optimization. By framing image-text retrieval as a series of preference choices (e.g., preferring that a neutral professional term maps equally well to different demographic groups), their method surgically alters the embedding space to decouple identity attributes from unrelated roles, ensuring that fairness does not come at the cost of performance.
Key Foundational Work
Image Captioning using Facial Expression and Attention (2020) — contributed 1 concept elements to this direction.
Explicit Bias Discovery in Visual Question Answering Models (2019) — contributed 1 concept elements to this direction.