Introduction
As Vision-Language Models (VLMs) become central to automated decision-making, understanding their internal biases is critical. While previous research has documented biased outputs, this study investigates the underlying reasoning mechanisms. The authors explore how race and gender compound—intersectional bias—to influence model behavior when predicting occupations from cropped facial images.
Methodology
The study evaluated five open-source VLMs: Gemma-3-4B, Qwen-2.5-VL (3B and 7B), and LLaVA-1.5 (7B and 13B). Using a curated subset of 420 samples from the FairFace dataset (30 samples for each of 14 intersectional race-gender combinations), the researchers tested three prompting styles:
- Direct Prediction: Label-only output.
- Direct Prediction with Reasoning: Label plus justification.
- Top-3 Ranking with Reasoning: Three likely labels and a combined rationale.
By using face-only crops, the models were forced to rely on internal associations between physical features and professional roles rather than environmental context.
The Illusion of Logic: Post-hoc Rationalization
A key finding is that VLMs often perform "reasoning" in reverse. Instead of deriving a label from features, they appear to select a label based on a stereotype and then "hallucinate" a visual justification. This creates a "logical veneer" that makes biased errors harder for human users to detect.
For example, the same "smiling" expression might be linked to "Engineer" for a man but "Homemaker" for a woman. Cultural markers like turbans or hijabs often serve as "shortcuts" for specific professions, while generic cues are used for groups lacking distinct cultural attire.
Intersectional Bias Trends
The distribution of predicted occupations reveals deep-seated intersectional skews. Biases were not uniform across gender or race alone but emerged at their intersection.
As shown in the cumulative counts, "Indian Females" were disproportionately categorized as Homemakers or Social Workers. Conversely, "Criminal" predictions were almost exclusively associated with male demographics, particularly Black and Indian males. While adding a reasoning step slightly reduced the intensity of some peaks, the overall skewed distributions remained largely unchanged across all tested models.
Conclusion
The act of reasoning does not inherently eliminate bias in VLMs; in some cases, it provides a sophisticated mask for it. Larger models show more nuance—identifying task-relevant cues like an "open mouth" for a musician—but still exhibit significant variance and stereotypical associations. Future work must focus on de-biasing the reasoning process itself to ensure models make decisions for the right reasons, rather than just producing the right labels.
Summary updated