ContextBias: Controlled Evaluation of Bias Persistence Under Context Shift in Text-to-Image Models

Shaghayegh Kolli, Sina Emami, Moreno D'Incà, Pouyan Nejadi, Nicu Sebe, Massimiliano Mancini, Jana DiesnerarXiv 2026 · arXiv

TL;DR

When text-to-image models are prompted to place professional roles in completely unrelated environments, they change the background but leave stereotypical role-linked attributes (like gender and attire) completely intact. To systematically study this phenomenon, the authors introduce ContextBias, a controlled evaluation framework showing that person-level demographic and occupational stereotypes are stubbornly context-immune across state-of-the-art models.

What Problem Does This Paper Address?

Most existing evaluations of stereotypical bias in text-to-image (T2I) models look at concepts in isolation—for example, generating images for "a photo of a doctor" and analyzing the resulting gender or racial distributions. However, real-world prompts are rarely context-free; users naturally place people within specific locations and activities (e.g., "a doctor exercising in a park").

This paper addresses a fundamental question about compositional generalization: Does changing the scene context reshape learned visual associations, or does it merely change the background around a rigidly biased representation of a person? Understanding whether these stereotypical associations persist or weaken under contextual shifts is crucial for evaluating model fairness and understanding how generative models visually define social roles.

Methods

The authors developed ContextBias, a framework designed to isolate the impact of context variation while keeping the core role identity fixed.

Figure 1: Our overall logic for testing whether role- linked cues adapt to scene context by varying the scene while fixing the role.

The workflow consists of three main stages:

  1. Controlled Prompt Generation: Prompts are constructed under three distinct conditions:
  2. Context-Free (CF): "A photo of a [role]" (baseline).
  3. Context-Aware Related (CA-R): The role is placed in a semantically congruent setting (e.g., a chef in a kitchen).
  4. Context-Aware Unrelated (CA-U): The role is placed in a semantically incongruent setting (e.g., a chef in a park).
  5. Image Generation: Images are generated across four state-of-the-art T2I models: FLUX.1, Stable Diffusion XL (SDXL), Stable Diffusion 3.5, and Qwen-Image.
  6. Schema-Guided Attribute Extraction: Instead of relying on raw text embeddings, the authors use a vision-language model (GPT-5-mini) as an automated auditor to extract 30 structured visual attributes across four cohorts: People, Camera, Objects, and Scene.

To quantify bias, they use Bias Intensity (BI) (based on Shannon entropy to measure distribution concentration) and the Context Consistency Score (CCS) (to measure how stably a specific label persists across different contexts).

Figure 2: ContextBias workflow. From a set of roles, an LLM constructs semantically matched prompts across three contextual conditions: context-free (CF), context-related (CA-R), and context-unrelated (CA-U). T2I models generate images for each prompt and seed. A schema-guided vision-language pipeline extracts structured visual attributes, which are aggregated, and calculates a Bias Intensity (BI) and the Context Consistency Score (CCS).

Datasets

The authors introduce ContextBench, a controlled prompt benchmark spanning:

  • 92 professional roles curated from the U.S. Bureau of Labor Statistics Standard Occupational Classification (SOC).
  • 1,656 semantically controlled prompts (18 configurations per role, including semantic paraphrases and location/activity substitutions generated via GPT-4o-mini and manually filtered).
  • 66,240 generated images (16,560 images per generator across 10 independent random seeds).

Key Insights and Ideas

  • Stereotypes are Context-Immune: Forcing a model to generate a role in an unrelated context does not suppress role-linked attributes. A flight attendant reading a book in a park is still overwhelmingly depicted with the same demographic markers and uniform elements as one in an airplane.
  • Unrelated Contexts Increase Distributional Concentration: Counter-intuitively, placing roles in unrelated contexts increases the overall benchmark-wide bias intensity (pooled BI increases by $+0.047$). Rather than neutralizing priors, unrelated contexts pull different roles toward a shared default representation.
  • Layered Image Adaptation: Models treat image composition hierarchically. Scene appearance and Camera framing are highly sensitive to contextual prompts, whereas People-level attributes (gender presentation, body type, characteristic garments) remain rigidly locked to the role's internal prototype.
Figure 3: Mean Bias Intensity (BI), pooled over roles, across contextual conditions (CF, CA-R, CA-U) for four attribute cohorts.

Key Results

  • Demographic Persistence: Gender cues show the strongest resistance to context. For instance, "Dancer" is generated as female in 98–100% of images across all four models ($\Delta = 1.0$ percentage points across contexts). "Mechanic" and "Carpenter" remain stubbornly male (94–100%) even when placed in completely unrelated scenes.
  • Garment and Tool Persistence: Occupational markers frequently persist outside their job context. Welders retain their protective gloves (29–32% prevalence) and bakers retain their aprons (31–33% prevalence) across all context conditions.
  • Robustness to Phrasing: The persistence patterns are not artifacts of specific prompt wordings. The evaluation shows that 93.3% of role-label tuples remain invariant across semantic paraphrases and prompt reformulations.
Figure 4: Mean Bias Intensity per dimension, pooled over roles and averaged across models and conditions. Color encodes attribute cohort: People, Camera, Ob- jects, Scene.

Limitations and Future Work

  • Context Scope: The framework currently focuses only on location and activity variations. Other critical contextual dimensions—such as lighting, cultural settings, or interpersonal interactions—are not yet explored.
  • Geocultural Bias: ContextBench is built around U.S. occupational categories and predominantly Western photographic contexts. The findings reflect representational patterns within English-language, Western-centric models and may not generalize globally.
  • Measurement vs. Harm: The Bias Intensity (BI) metric measures statistical concentration, not explicit social harm. A highly concentrated distribution might reflect a genuine visual reality of an occupation rather than a harmful stereotype; distinguishing between the two requires normative judgments outside the scope of this framework.

Summary updated

Abstract

Text-to-image models learn associations between concepts - in the case of this paper, people's professions, which we refer to as roles - and visual attributes. These associations can underpin many observed forms of stereotypical bias. A key open question in this area is whether these associations are stable or change when visual representations of people in professional roles are placed in different prompted contexts. We introduce ContextBias, a controlled evaluation framework, and ContextBench, a benchmark spanning 92 roles and 1,656 semantically controlled prompts, designed to isolate the effect of contextual variation on role-linked visual representations. Evaluating four state-of-the-art models on 66,240 generated images, we find that placing a role in a semantically unrelated context does not suppress role-linked attributes; instead, cross-role attribute concentration increases (pooled BI $+0.047$). Demographic cues, characteristic garments, and role-specific tools remain highly prevalent across context-free, related, and unrelated conditions, and are robust to semantic prompt reformulation. Scene composition and camera framing show the greatest context-sensitivity. These findings reveal a form of stereotypical persistence that remains largely invisible to context-free evaluations, highlighting the need for controlled contextual variation in bias benchmarking. Code and dataset: https://huggingface.co/datasets/shaghayegh/ContextBias , https://github.com/Sina-Emami/ContextBias
BibTeX
@misc{kolli2026contextbias,
  author = {Shaghayegh Kolli and Sina Emami and Moreno D'Incà and Pouyan Nejadi and Nicu Sebe and Massimiliano Mancini and Jana Diesner},
  title = {{ContextBias: Controlled Evaluation of Bias Persistence Under Context Shift in Text-to-Image Models}},
  year = {2026},
  eprint = {2608.29847},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.29847},
}
Figure