TL;DR
When text-to-image models are prompted to place professional roles in completely unrelated environments, they change the background but leave stereotypical role-linked attributes (like gender and attire) completely intact. To systematically study this phenomenon, the authors introduce ContextBias, a controlled evaluation framework showing that person-level demographic and occupational stereotypes are stubbornly context-immune across state-of-the-art models.
What Problem Does This Paper Address?
Most existing evaluations of stereotypical bias in text-to-image (T2I) models look at concepts in isolation—for example, generating images for "a photo of a doctor" and analyzing the resulting gender or racial distributions. However, real-world prompts are rarely context-free; users naturally place people within specific locations and activities (e.g., "a doctor exercising in a park").
This paper addresses a fundamental question about compositional generalization: Does changing the scene context reshape learned visual associations, or does it merely change the background around a rigidly biased representation of a person? Understanding whether these stereotypical associations persist or weaken under contextual shifts is crucial for evaluating model fairness and understanding how generative models visually define social roles.
Methods
The authors developed ContextBias, a framework designed to isolate the impact of context variation while keeping the core role identity fixed.
The workflow consists of three main stages:
- Controlled Prompt Generation: Prompts are constructed under three distinct conditions:
- Context-Free (CF): "A photo of a [role]" (baseline).
- Context-Aware Related (CA-R): The role is placed in a semantically congruent setting (e.g., a chef in a kitchen).
- Context-Aware Unrelated (CA-U): The role is placed in a semantically incongruent setting (e.g., a chef in a park).
- Image Generation: Images are generated across four state-of-the-art T2I models: FLUX.1, Stable Diffusion XL (SDXL), Stable Diffusion 3.5, and Qwen-Image.
- Schema-Guided Attribute Extraction: Instead of relying on raw text embeddings, the authors use a vision-language model (GPT-5-mini) as an automated auditor to extract 30 structured visual attributes across four cohorts: People, Camera, Objects, and Scene.
To quantify bias, they use Bias Intensity (BI) (based on Shannon entropy to measure distribution concentration) and the Context Consistency Score (CCS) (to measure how stably a specific label persists across different contexts).
Datasets
The authors introduce ContextBench, a controlled prompt benchmark spanning:
- 92 professional roles curated from the U.S. Bureau of Labor Statistics Standard Occupational Classification (SOC).
- 1,656 semantically controlled prompts (18 configurations per role, including semantic paraphrases and location/activity substitutions generated via GPT-4o-mini and manually filtered).
- 66,240 generated images (16,560 images per generator across 10 independent random seeds).
Key Insights and Ideas
- Stereotypes are Context-Immune: Forcing a model to generate a role in an unrelated context does not suppress role-linked attributes. A flight attendant reading a book in a park is still overwhelmingly depicted with the same demographic markers and uniform elements as one in an airplane.
- Unrelated Contexts Increase Distributional Concentration: Counter-intuitively, placing roles in unrelated contexts increases the overall benchmark-wide bias intensity (pooled BI increases by $+0.047$). Rather than neutralizing priors, unrelated contexts pull different roles toward a shared default representation.
- Layered Image Adaptation: Models treat image composition hierarchically. Scene appearance and Camera framing are highly sensitive to contextual prompts, whereas People-level attributes (gender presentation, body type, characteristic garments) remain rigidly locked to the role's internal prototype.
Key Results
- Demographic Persistence: Gender cues show the strongest resistance to context. For instance, "Dancer" is generated as female in 98–100% of images across all four models ($\Delta = 1.0$ percentage points across contexts). "Mechanic" and "Carpenter" remain stubbornly male (94–100%) even when placed in completely unrelated scenes.
- Garment and Tool Persistence: Occupational markers frequently persist outside their job context. Welders retain their protective gloves (29–32% prevalence) and bakers retain their aprons (31–33% prevalence) across all context conditions.
- Robustness to Phrasing: The persistence patterns are not artifacts of specific prompt wordings. The evaluation shows that 93.3% of role-label tuples remain invariant across semantic paraphrases and prompt reformulations.
Limitations and Future Work
- Context Scope: The framework currently focuses only on location and activity variations. Other critical contextual dimensions—such as lighting, cultural settings, or interpersonal interactions—are not yet explored.
- Geocultural Bias: ContextBench is built around U.S. occupational categories and predominantly Western photographic contexts. The findings reflect representational patterns within English-language, Western-centric models and may not generalize globally.
- Measurement vs. Harm: The Bias Intensity (BI) metric measures statistical concentration, not explicit social harm. A highly concentrated distribution might reflect a genuine visual reality of an occupation rather than a harmful stereotype; distinguishing between the two requires normative judgments outside the scope of this framework.
Summary updated