RS-Corrector: Correcting the Racial Stereotypes in Latent Diffusion Models

Yue Jiang, Yueming Lyu, Tianxiang Ma, Bo Peng, Jing DongarXiv 2023 · DOI · arXiv

Abstract

Large-scale text-to-image models like Stable Diffusion often inherit racial stereotypes from their training data, leading to distorted or defamatory representations. We present RS-Corrector, a plug-and-play inference-stage framework that steers the generation process toward fairer, more aesthetically pleasing results without requiring expensive retraining or fine-tuning. By establishing an anti-stereotypical latent prior and applying strategic guidance during the later stages of denoising, RS-Corrector significantly improves facial aesthetics and attribute fairness across diverse demographic groups.

Introduction

Text-to-image (T2I) models have revolutionized creative workflows, but their reliance on massive, uncurated datasets like LAION-5B introduces significant ethical challenges. These models frequently exhibit "intrinsic stereotypes"—biases baked into the model weights that manifest as exaggerated facial features or age inconsistencies for specific racial groups. For instance, prompts for Asian faces often yield images with disproportionately small eyes and high cheekbones, while Indian and Latin faces are frequently depicted as significantly older than their White counterparts for the same age-neutral prompt. These biases are difficult to remove via traditional fine-tuning without compromising the model's creative capabilities.

The RS-Corrector Framework

RS-Corrector addresses these biases during the inference stage, avoiding the prohibitive cost of fine-tuning multi-billion parameter models. The framework employs a two-pronged strategy to "nudge" the latent code away from stereotypical regions of the latent space:

  1. Anti-Stereotypical Latent Prior ($L_{tc}$): A simple projector is trained using a contrastive learning approach. It is fed "positive" samples (fair, high-quality images from a curated dataset) and "negative" samples (stereotypical images generated by the original Stable Diffusion). This teaches the projector to distinguish between fair and biased representations.
  2. Direct Anti-Stereotypical Guidance ($L_c$): During the denoising process, the model uses this learned prior to calculate a loss, updating the latent code to maximize similarity with fair samples.
  3. Strategic Timing: The authors found that applying this correction in the later stages of the diffusion process is most effective. Since the core structure is finalized early, later intervention allows the corrector to refine identity-defining facial features without disrupting the overall composition.
Detailed view of Asian face generation: Stable Diffusion (top) vs. RS-Corrector (bottom), showing the correction of exaggerated features mentioned in the text.

Experimental Setup

To evaluate the framework, the researchers curated a balanced dataset from LAION-5B, ensuring equal distribution of race (White, Black, Asian, Latin, and Indian), gender, and age to prevent "attribute leakage." Evaluation utilized objective metrics from the Face++ API and Deepface, focusing on facial aesthetics (beauty scores, skin health) and demographic accuracy.

Results and Discussion

RS-Corrector demonstrates substantial improvements over the baseline Stable Diffusion and other adaptation methods like Textual Inversion and DreamBooth.

  • Facial Aesthetics: For the Asian demographic, beauty scores increased from 54.45 to 62.04. Skin health scores also saw significant improvements across all groups.
  • Attribute Fairness: The model successfully mitigated age bias. While the original model often depicted Indian and Latin faces as older, RS-Corrector achieved much better age consistency across groups.
  • User Preference: In a study with 25 participants, images generated by RS-Corrector were preferred 61.28% of the time, compared to just 10.88% for the original Stable Diffusion.
  • Efficiency: The method adds minimal overhead, increasing inference time by only ~1.7 seconds (7.9s vs 6.2s for SD), making it significantly faster than fine-tuning-based methods like DreamBooth (~29.6s).
Comprehensive comparison of Stable Diffusion (top row) vs. RS-Corrector (bottom row) across five demographic groups, illustrating improvements in aesthetics and fairness.

Conclusion and Future Work

RS-Corrector provides a viable, efficient path toward fairer AI-generated imagery. By intervening at the latent level during inference, it corrects deep-seated stereotypes while preserving the model's creative flexibility. However, the authors note that cultural artifacts (e.g., facial paint in Indian groups) can still interfere with objective scoring. Future work will aim to expand this approach to non-facial biases, such as occupational or gender roles, and a broader range of intersectional identities.

Summary updated

Abstract

Recent text-conditioned image generation models have demonstrated an exceptional capacity to produce diverse and creative imagery with high visual quality. However, when pre-trained on billion-sized datasets randomly collected from the Internet, where potential biased human preferences exist, these models tend to produce images with common and recurring stereotypes, particularly for certain racial groups. In this paper, we conduct an initial analysis of the publicly available Stable Diffusion model and its derivatives, highlighting the presence of racial stereotypes. These models often generate distorted or biased images for certain racial groups, emphasizing stereotypical characteristics. To address these issues, we propose a framework called "RS-Corrector", designed to establish an anti-stereotypical preference in the latent space and update the latent code for refined generated results. The correction process occurs during the inference stage without requiring fine-tuning of the original model. Extensive empirical evaluations demonstrate that the introduced \themodel effectively corrects the racial stereotypes of the well-trained Stable Diffusion model while leaving the original model unchanged.
BibTeX
@misc{jiang2023rs,
  author = {Yue Jiang and Yueming Lyu and Tianxiang Ma and Bo Peng and Jing Dong},
  title = {{RS-Corrector: Correcting the Racial Stereotypes in Latent Diffusion Models}},
  year = {2023},
  eprint = {2312.04810},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.04810},
}
Figure