How Reasoning Influences Intersectional Biases in Vision Language Models

Adit Desai, Sudipta Roy, Mohna ChakrabortyarXiv 2025 · DOI · arXiv

Introduction

As Vision-Language Models (VLMs) become central to automated decision-making, understanding their internal biases is critical. While previous research has documented biased outputs, this study investigates the underlying reasoning mechanisms. The authors explore how race and gender compound—intersectional bias—to influence model behavior when predicting occupations from cropped facial images.

Methodology

The study evaluated five open-source VLMs: Gemma-3-4B, Qwen-2.5-VL (3B and 7B), and LLaVA-1.5 (7B and 13B). Using a curated subset of 420 samples from the FairFace dataset (30 samples for each of 14 intersectional race-gender combinations), the researchers tested three prompting styles:

  1. Direct Prediction: Label-only output.
  2. Direct Prediction with Reasoning: Label plus justification.
  3. Top-3 Ranking with Reasoning: Three likely labels and a combined rationale.

By using face-only crops, the models were forced to rely on internal associations between physical features and professional roles rather than environmental context.

The Illusion of Logic: Post-hoc Rationalization

A key finding is that VLMs often perform "reasoning" in reverse. Instead of deriving a label from features, they appear to select a label based on a stereotype and then "hallucinate" a visual justification. This creates a "logical veneer" that makes biased errors harder for human users to detect.

A pink shirt used as a visual cue for domesticity.

For example, the same "smiling" expression might be linked to "Engineer" for a man but "Homemaker" for a woman. Cultural markers like turbans or hijabs often serve as "shortcuts" for specific professions, while generic cues are used for groups lacking distinct cultural attire.

The distribution of predicted occupations reveals deep-seated intersectional skews. Biases were not uniform across gender or race alone but emerged at their intersection.

Cumulative counts for Homemaker, Criminal, Social Worker, and Business Owner across demographic groups and models.

As shown in the cumulative counts, "Indian Females" were disproportionately categorized as Homemakers or Social Workers. Conversely, "Criminal" predictions were almost exclusively associated with male demographics, particularly Black and Indian males. While adding a reasoning step slightly reduced the intensity of some peaks, the overall skewed distributions remained largely unchanged across all tested models.

Conclusion

The act of reasoning does not inherently eliminate bias in VLMs; in some cases, it provides a sophisticated mask for it. Larger models show more nuance—identifying task-relevant cues like an "open mouth" for a musician—but still exhibit significant variance and stereotypical associations. Future work must focus on de-biasing the reasoning process itself to ensure models make decisions for the right reasons, rather than just producing the right labels.

Summary updated

BibTeX
@misc{desai2025how,
  author = {Adit Desai and Sudipta Roy and Mohna Chakraborty},
  title = {{How Reasoning Influences Intersectional Biases in Vision Language Models}},
  year = {2025},
  eprint = {2511.06005},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.06005},
}
Figure