Abstract
As Vision-Language Models (VLMs) become increasingly integrated into real-world applications, understanding their latent social biases is critical. This paper introduces VIGNETTE, a massive benchmark comprising over 30 million synthetic images designed to evaluate VLM bias across four socially grounded dimensions: factuality, perception, stereotyping, and decision-making. Our evaluation of state-of-the-art models reveals that biases are deeply contextual, often favoring dominant social groups while penalizing marginalized identities through subtle, non-obvious mechanisms.
1. Introduction
Research into bias within Large Language Models (LLMs) has matured significantly, yet our understanding of Vision-Language Models (VLMs) remains limited. Existing benchmarks often rely on "decontextualized" imagery—such as simple portraits—and focus narrowly on gender-occupation tropes (e.g., associating men with medicine and women with nursing). However, in real-world scenarios, stereotypes are frequently triggered by activity and context. VIGNETTE addresses this gap by providing a framework to probe how models make latent inferences about capability, trustworthiness, and social status based on visual cues in grounded settings.
2. The VIGNETTE Framework
The researchers developed a Visual Question Answering (VQA) framework that evaluates models through four distinct paradigms, moving beyond simple identification to complex social reasoning.
- Factuality: Tests the model's ability to correctly identify individuals and their actions.
- Perception: Examines implicit assumptions regarding a person's capabilities or preferences (e.g., "Who is struggling with this task?").
- Stereotyping: Probes the association of specific traits (e.g., honesty, competence) with identities in the absence of activity-based evidence.
- Decision-Making: Evaluates how these biases manifest in downstream tasks, such as hiring or promotion scenarios.
3. Methodology and Dataset
The VIGNETTE benchmark utilizes a staggering collection of 30M+ synthetic images generated via the FLUX.1-dev model. The dataset covers 167 unique identities across eight dimensions, including ability, age, gender, nationality, religion, and socioeconomic status. These identities are paired with 75 diverse activities categorized by time-use.
To measure bias, the study employs a pairwise evaluation setup. By presenting two individuals side-by-side, the model is forced to make relative judgments. Metrics such as Log-Odds Ratios and PairComp are used to quantify disproportionate selection and determine how the presence of one identity shifts the perception of another.
4. Key Insights and Results
Evaluation of models like LLaVA-1.6-7B, Llama-3.2-11B-Vision-Instruct, and DeepSeek-VL2-4.5B yielded several critical findings:
- The "Dominance" Advantage: Models demonstrate significantly higher factual accuracy for socially dominant identities (e.g., "athletic," "healthy," Western nationalities). Marginalized groups are more likely to be misidentified, even when visual markers are explicit.
- Relative Perception: A model's perception of an identity is not static. An individual may be viewed as capable when paired with one group but as "struggling" when paired with a more "dominant" group, reflecting encoded global power hierarchies.
- The Vision "Tax": Comparing text-only prompts to multimodal prompts revealed that the visual component specifically increases the selection of Europeans while biasing against Asians, suggesting the vision encoder itself is a primary source of Western-centric bias.
5. Discussion and Conclusion
While VIGNETTE provides a robust foundation for evaluating VLM bias, it is not without limitations. The use of synthetic images, while necessary for scale and control, may not perfectly mirror the nuances of real-world photography. Furthermore, certain "invisible" identities (e.g., mental health status) were excluded to avoid relying on harmful visual tropes for representation.
In conclusion, VIGNETTE reveals that VLM biases are multi-layered and often contradictory. Models do not simply possess a uniform "pro-Western" bias; they encode complex stereotypes that reward or penalize groups differently based on the specific task. This benchmark serves as a vital tool for future "red-teaming" and the development of alignment techniques that move beyond simple text-based safety filters toward true social grounding.
Summary updated