Text-to-image (T2I) diffusion models like Stable Diffusion achieve remarkable generation quality, but they are notorious for inheriting and amplifying biases from their training data. Even when trained on relatively balanced datasets, these models often gravitate toward a few "majority" representations—for instance, generating predominantly white, male figures when prompted for a "doctor."
Historically, researchers have tackled this bias in two ways. Closed-set approaches mitigate biases in predefined categories (like gender or race), but require researchers to know exactly what they are looking for beforehand. Open-set approaches, on the other hand, identify what the model over-produces (the majority attributes) and attempt to suppress them.
However, both strategies overlook a critical piece of the puzzle: minority attributes. These are features—such as specific cultural symbols or rare occupations—that are encoded in the model's internal representations but are systematically suppressed during normal generation.
To address this, researchers have introduced RAIGen, the first label-free framework designed to discover these rare attributes without relying on predefined categories.
The "Whac-A-Mole" Dilemma and Model Scaling
A core insight driving RAIGen is the "Whac-A-Mole" dilemma in AI debiasing. The researchers demonstrate that simply suppressing a dominant trait (e.g., "white race") does not lead to uniform diversity. Instead, the model's probability mass often collapses onto another single minority group, leaving other groups completely underrepresented.
Crucially, RAIGen proves that models often know about these rare attributes—they are mathematically encoded in the model's weights—but the sampling process suppresses them. This suggests that achieving diversity might not always require expensive model retraining; rather, it requires better navigation of the model's existing latent space.
Interestingly, the study also found that scaling up doesn't necessarily solve bias. Larger, higher-capacity models like SDXL actually exhibited a lower presence of rare attributes in certain categories compared to the older Stable Diffusion v1.4, suggesting that model scaling can inadvertently lead to "contextual homogenization."
How RAIGen Works
RAIGen operates directly on the internal "bottleneck" representations of diffusion models through a three-stage pipeline:
- Feature Decomposition via Matryoshka Sparse Autoencoders (MSAEs): Standard Sparse Autoencoders break down complex neural activations into human-interpretable concepts. RAIGen uses a hierarchical "Matryoshka" version, allowing researchers to focus on "coarse" levels that capture broad, stable semantics (like "female doctor") rather than "fine" levels that might just capture random pixel patterns.
- The Minority Score: To identify which neurons represent rare attributes, RAIGen introduces a novel metric that balances two signals:
- Rarity: How infrequently the neuron fires across thousands of generated images.
-
Semantic Distinctiveness: How different the images triggering this neuron are compared to the model's "average" output. This ensures the framework finds meaningful, distinct concepts rather than just "dead" or noisy neurons.
-
Annotation and Amplification: Once rare neurons are identified, Multimodal Large Language Models (MLLMs) are used to describe them (e.g., "doctor with a medical chart in the background"). These descriptions can then be fed back into the model via prompt revision, forcing the model to generate these rare, hidden modes.
Key Results and Efficiency
The researchers evaluated RAIGen across multiple datasets, including WinoBias (for occupational biases) and COCO (for everyday scenes).
- Broad Compatibility: RAIGen successfully identified rare attributes across multiple architectures, including Stable Diffusion (v1.4, v2.1, XL) and the transformer-based FLUX.1-schnell.
- Effective Amplification: By using RAIGen to revise prompts, the researchers significantly reduced output skew. For SDXL on the WinoBias dataset, the deviation from a perfectly balanced distribution dropped dramatically from 0.49 to 0.23.
- High Efficiency: The entire discovery pipeline for a single prompt takes about 72 minutes on a single A100 GPU, with the bulk of that time spent on initial image generation rather than the analysis itself.
Limitations and Future Directions
While RAIGen represents a significant step forward in label-free bias auditing, it has limitations. Most notably, it can only uncover what the model already knows; if a model never learned about a specific cultural group during training, RAIGen cannot hallucinate it.
Additionally, the current MSAE must be trained or fine-tuned for specific prompts. Future work aims to develop "universal" SAEs across a wide range of prompts to reduce compute overhead. Finally, while RAIGen works on newer Diffusion Transformer (DiT) architectures like FLUX, their features are less spatially grounded than traditional U-Net models, making them harder to visualize. Finding better "hook points" for feature extraction in DiTs remains an active area of research.
Summary updated