Text-to-image (T2I) models like Stable Diffusion and DALL-E have revolutionized content creation but often inherit societal biases from their training data. When prompted with neutral terms like "doctor" or "CEO," these models frequently over-represent specific demographics—typically Western, light-skinned males. This research explores a novel, human-centric approach to mitigation: Ethical Natural Language Interventions.
The ENTIGEN Framework
Instead of complex fine-tuning or post-processing, the authors investigate whether simply appending abstract ethical advice to a prompt can diversify outputs. They introduced the ENTIGEN (Ethical NaTural Language Interventions in Text-to-Image GENeration) benchmark to test this hypothesis across three primary axes: gender, skin color, and culture.
As shown in Figure 1, the framework compares "Original" prompts (e.g., "a photo of a bride") against "Intervened" prompts (e.g., "a photo of a bride from diverse cultures"). The study evaluates whether the zero-shot capabilities of modern T2I models are sophisticated enough to act upon these abstract social concepts.
Methodology
The researchers utilized a hybrid evaluation strategy to measure the effectiveness of these interventions:
- Prompt Engineering: Interventions were either axis-specific or "irrelevant" (e.g., "...if honesty is the best policy") to serve as a control.
- Diversity Score (DS): A metric used to measure how evenly the model distributes generations across different social groups. A lower score indicates higher diversity.
- Hybrid Assessment: To ensure accuracy, the team combined CLIP-based automated evaluation with human evaluation via Amazon MTurk.
Key Results
The study found that T2I models can indeed follow ethical instructions to reduce bias without sacrificing image quality. The phrase "irrespective of [axis]" acted as a powerful catalyst for diversity, likely due to its association with meritocracy in the models' pre-training data.
As detailed in Figure 2, the impact was significant across multiple models:
- Gender Diversity: In Stable Diffusion, the human-rated Gender Diversity Score improved from 0.82 to 0.68 with the intervention.
- Skin Color: Interventions successfully reduced bias, though the effect varied by model.
- Cultural Representation: The most dramatic shifts occurred in cultural prompts. For "a photo of a bride," the DS dropped from 0.92 (highly Western-biased) to 0.33 (highly diverse) when the model was instructed to generate "from diverse cultures."
Insights and Limitations
While effective, the researchers noted that interventions are not a "catch-all" solution. A prompt designed to fix gender bias does not automatically address skin color bias, necessitating targeted interventions for each axis. Furthermore, the study highlighted that while CLIP is reliable for gender evaluation (78-85% human agreement), it struggles with skin color (67-78%), reinforcing the need for human-in-the-loop assessments.
Future work aims to move beyond binary categories (man/woman, light/dark skin) to include non-binary identities and a broader spectrum of skin tones, such as the Fitzpatrick scale, while addressing the geographic biases inherent in current human evaluation pools.
Summary updated