Abstract
Large-scale text-to-image models like Stable Diffusion often inherit racial stereotypes from their training data, leading to distorted or defamatory representations. We present RS-Corrector, a plug-and-play inference-stage framework that steers the generation process toward fairer, more aesthetically pleasing results without requiring expensive retraining or fine-tuning. By establishing an anti-stereotypical latent prior and applying strategic guidance during the later stages of denoising, RS-Corrector significantly improves facial aesthetics and attribute fairness across diverse demographic groups.
Introduction
Text-to-image (T2I) models have revolutionized creative workflows, but their reliance on massive, uncurated datasets like LAION-5B introduces significant ethical challenges. These models frequently exhibit "intrinsic stereotypes"—biases baked into the model weights that manifest as exaggerated facial features or age inconsistencies for specific racial groups. For instance, prompts for Asian faces often yield images with disproportionately small eyes and high cheekbones, while Indian and Latin faces are frequently depicted as significantly older than their White counterparts for the same age-neutral prompt. These biases are difficult to remove via traditional fine-tuning without compromising the model's creative capabilities.
The RS-Corrector Framework
RS-Corrector addresses these biases during the inference stage, avoiding the prohibitive cost of fine-tuning multi-billion parameter models. The framework employs a two-pronged strategy to "nudge" the latent code away from stereotypical regions of the latent space:
- Anti-Stereotypical Latent Prior ($L_{tc}$): A simple projector is trained using a contrastive learning approach. It is fed "positive" samples (fair, high-quality images from a curated dataset) and "negative" samples (stereotypical images generated by the original Stable Diffusion). This teaches the projector to distinguish between fair and biased representations.
- Direct Anti-Stereotypical Guidance ($L_c$): During the denoising process, the model uses this learned prior to calculate a loss, updating the latent code to maximize similarity with fair samples.
- Strategic Timing: The authors found that applying this correction in the later stages of the diffusion process is most effective. Since the core structure is finalized early, later intervention allows the corrector to refine identity-defining facial features without disrupting the overall composition.
Experimental Setup
To evaluate the framework, the researchers curated a balanced dataset from LAION-5B, ensuring equal distribution of race (White, Black, Asian, Latin, and Indian), gender, and age to prevent "attribute leakage." Evaluation utilized objective metrics from the Face++ API and Deepface, focusing on facial aesthetics (beauty scores, skin health) and demographic accuracy.
Results and Discussion
RS-Corrector demonstrates substantial improvements over the baseline Stable Diffusion and other adaptation methods like Textual Inversion and DreamBooth.
- Facial Aesthetics: For the Asian demographic, beauty scores increased from 54.45 to 62.04. Skin health scores also saw significant improvements across all groups.
- Attribute Fairness: The model successfully mitigated age bias. While the original model often depicted Indian and Latin faces as older, RS-Corrector achieved much better age consistency across groups.
- User Preference: In a study with 25 participants, images generated by RS-Corrector were preferred 61.28% of the time, compared to just 10.88% for the original Stable Diffusion.
- Efficiency: The method adds minimal overhead, increasing inference time by only ~1.7 seconds (7.9s vs 6.2s for SD), making it significantly faster than fine-tuning-based methods like DreamBooth (~29.6s).
Conclusion and Future Work
RS-Corrector provides a viable, efficient path toward fairer AI-generated imagery. By intervening at the latent level during inference, it corrects deep-seated stereotypes while preserving the model's creative flexibility. However, the authors note that cultural artifacts (e.g., facial paint in Indian groups) can still interfere with objective scoring. Future work will aim to expand this approach to non-facial biases, such as occupational or gender roles, and a broader range of intersectional identities.
Summary updated