Portrait Fitness as a Pretext: Probing Implicit Demographic Bias in VLMs

Alexandre Fournier-Montgieux, Akram Elbouanani, Aboubacar Tuo, Adrian Popescu, Hervé Le Borgne, Bertrand LuvisonarXiv 2026 · arXiv

TL;DR

Researchers have developed BALPORTRAITS, a controlled dataset of 9,000 AI-generated portraits, to audit implicit demographic bias in Vision-Language Models (VLMs). By using an "advertising-casting" pretext, they discovered that all 16 tested models—including frontier models like GPT-4o mini and Claude 3.5 Haiku—exhibit significant, context-dependent biases that often only surface at the intersection of age, gender, and ethnicity.

What Problem Does This Paper Address?

Measuring bias in modern VLMs is a game of cat and mouse. Direct questions about sensitive attributes (e.g., "Is this person likely to be a CEO?") often trigger safety filters and refusals. Furthermore, using "in-the-wild" images introduces confounding variables like clothing, background, and image quality, making it hard to isolate whether a model's judgment is based on a person's face or their surroundings.

The authors address this by shifting from direct classification to implicit association. They ask models to perform a "pretext" task: scoring how "fit" a portrait is for various advertising scenarios. This bypasses refusal mechanisms while revealing the underlying social hierarchies the models have internalized.

Methods

The core of the study is a "fitness-as-pretext" framework. Instead of asking for a person's profession, the researchers ask the VLM to score (from 1 to 10) how suitable a portrait is for an advertisement for a specific occupation, brand, or financial product.

Figure 2. Overview of the BALPORTRAITS curation and VLM face bias analysis pipeline.

To turn these raw scores into a meaningful metric of bias, they:

  1. Standardized Scenarios: Evaluated 100 occupations, 50 brands, 50 activities, and 50 banking/insurance products.
  2. Prestige Weighting: Assigned "prestige scores" to each item (e.g., "Chief Executive" has higher prestige than "Dishwasher"). These scores were validated by human panels.
  3. Rank-Based Analysis: Calculated a Degree of Bias (DoB), defined as the standard deviation of mean ranks across 24 demographic segments. This measures how much the model's "desirability" rankings deviate from a state of demographic equity.

Datasets

The authors introduced BALPORTRAITS, a dataset of 9,000 fictional portraits generated using gpt-image-1.

  • Balance: It covers 24 segments: 4 ethnicities (African, Asian, Caucasian, Indian) × 2 genders × 3 age anchors (25, 40, and 55 years old).
  • Control: All portraits use identical framing, neutral expressions, black t-shirts, and simple backgrounds to eliminate visual confounders.
  • Privacy: Because the faces are fictional and passed identity-change verification against their real-world "seeds," the dataset avoids the privacy pitfalls of using real people's photos.

Key Insights and Ideas

  • Bias is a Chameleon: A model might seem fair in one scenario but highly biased in another. For example, ethnicity is the dominant factor in brand rankings, while age is the primary driver of bias in banking and activity scenarios.
  • The Intersectional "Aha": Single-attribute audits (just gender or just race) are insufficient. Many disparities only emerge at the intersection. For instance, in banking, the "female penalty" often widens significantly as the age of the portrait increases.
  • Scaling Doesn't Solve Bias: Surprisingly, within model families, larger and more "capable" models often exhibited higher degrees of demographic dispersion than their smaller counterparts.
  • The "Caucasian" Baseline: Across almost all models and scenarios, Caucasian portraits were consistently ranked in the top two ethnic segments and were never the most penalized group. Conversely, Indian portraits were systematically discriminated against across nearly all tasks.
Figure 4. One-way demographic effects across scenarios for selected VLMs. Each subfigure reports mean portrait ranks by ethnicity, age, or gender, aggregated with scenario-level prestige scores. Lower ranks indicate higher prestige-weighted fitness. The figure shows that the dominant demographic dimension varies by scenario: ethnicity is more visible in brand rankings, while age is more visible in activities and banking/insurance.

Key Results

The audit of 16 VLMs yielded sobering results:

  • Universal Bias: Every single model tested showed demographic dispersion significantly above the "equity line" (the level of variation expected by chance).
  • Statistical Significance: On average, 73% of all pairwise segment comparisons (e.g., 25-year-old Asian Female vs. 55-year-old Caucasian Male) were statistically significant.
  • Frontier Model Performance: Even highly aligned models like GPT54M (GPT-4o mini) and CLAH45 (Claude 3.5 Haiku) showed high bias, with CLAH45 being among the most dispersed models in the study.

Limitations and Future Work

The authors acknowledge several limitations:

  • Fictional Identities: While generated portraits provide control, they aren't "counterfactual edits" (changing one attribute of the exact same face), meaning some facial geometry cues might still be entangled with demographic labels.
  • Broad Proxies: The four ethnic segments are broad appearance proxies and do not represent the full spectrum of human diversity or non-binary gender presentations.
  • Prestige Subjectivity: Prestige is culturally dependent; the study primarily uses Western-centric brand and occupation lists.

Future work aims to extend this to more diverse cultural contexts and evaluate how these implicit biases translate into real-world harms in deployed recommendation and screening systems.

The data and code are available at https://github.com/afm215/BALPortraits.

Summary updated

Abstract

We audit face-related bias in VLMs using BALPORTRAITS, a new balanced dataset of 9,000 fictional portraits across ethnicity, gender, and age segments. Sixteen VLMs estimate portrait fitness scores for occupations, brands, activities, and banking or insurance products within an advertising-casting framework designed to avoid direct inference of protected attributes. Item-level estimations are aggregated with scenario-specific prestige scores to compare rankings across demographic groups. While the magnitude and direction of these biases vary across scenarios, model families, and demographic dimensions, with age, ethnicity, and gender contributing differently depending on the task, we find demographic dispersion across all evaluated models, with a majority of pairwise segment differences reaching statistical significance. This study introduces a controlled protocol to assess implicit face-related associations and demonstrates that single-scenario or single-attribute audits are insufficient.
BibTeX
@misc{fourniermontgieux2026portrait,
  author = {Alexandre Fournier-Montgieux and Akram Elbouanani and Aboubacar Tuo and Adrian Popescu and Hervé Le Borgne and Bertrand Luvison},
  title = {{Portrait Fitness as a Pretext: Probing Implicit Demographic Bias in VLMs}},
  year = {2026},
  eprint = {2611.21403},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21403},
}
Figure