Abstract
Ethical issues around text-to-image (T2I) models demand a comprehensive control over the generative content. Existing techniques addressing these issues for responsible T2I models aim for the generated content to be fair and safe (non-violent/explicit). However, these methods remain bounded to handling the facets of responsibility concepts individually, while also lacking in interpretability. Moreover, they often require alteration to the original model, which compromises the model performance. In this work, we propose a unique technique to enable responsible T2I generation by simultaneously accounting for an extensive range of concepts for fair and safe content generation in a scalable manner. The key idea is to distill the target T2I pipeline with an external plug-and-play mechanism that learns an interpretable composite responsible space for the desired concepts, conditioned on the target T2I pipeline. We use knowledge distillation and concept whitening to enable this. At inference, the learned space is utilized to modulate the generative content. A typical T2I pipeline presents two plug-in points for our approach, namely; the text embedding space and the diffusion model latent space. We develop modules for both points and show the effectiveness of our approach with a range of strong results.
Introduction
State-of-the-art text-to-image (T2I) models, such as Stable Diffusion, are trained on massive, uncurated datasets harvested from the web. Consequently, these models often inherit and amplify societal biases—such as generating only white male subjects for the prompt "CEO"—or produce unsafe content, including violent or explicit imagery.
Existing solutions typically fall into three categories: prompt filtering, which is easily bypassed; "unlearning" concepts via model retraining, which is computationally expensive and can degrade image quality; or post-hoc moderation. Most of these methods lack interpretability and struggle to handle multiple "responsibility" facets (like race, age, and safety) simultaneously. This research introduces a scalable, modular framework to inject responsibility into T2I pipelines without altering the underlying frozen model.
Methodology: The Dual-Space Approach
The proposed framework utilizes a distillation-based approach that operates at two distinct stages of the T2I pipeline:
1. RICE: Responsible and Interpretable CLIP Embedding
This module targets the text encoder. Using knowledge distillation, the system learns a "responsible" version of the text embedding space. To ensure interpretability, the authors apply concept whitening, which decorrelates different semantic features. This allows users to adjust specific "knobs" (e.g., increasing racial diversity) without unintentionally affecting unrelated attributes.
2. RIIDL: Responsible Interpretable Intermediate Diffusion Latents
This module targets the U-Net, the component responsible for the iterative denoising process. RIIDL learns to inject responsibility signals into the early stages of generation. Research indicates that the high-level structure and semantic identity of an image are formed during the first 30% of the diffusion steps, making this the optimal window for intervention.
3. Dual-Space Integration
By combining RICE and RIIDL, the system manages both the global semantic context of the prompt and the fine-grained visual details during generation. Because these are "plug-and-play" add-ons, the original Stable Diffusion model remains unchanged, preserving its high-quality generation capabilities.
Experimental Evaluation
The framework was evaluated using several standard benchmarks:
- WinoBias: A dataset of 35 professions used to measure gender and racial representation.
- I2P (Inappropriate Image Prompts): A benchmark of over 4,500 prompts designed to trigger inappropriate content.
- Challenging Prompts: A subset of prompts known to produce heavily skewed results in standard models.
Key Results
The dual-space approach consistently outperformed or matched state-of-the-art specialized models:
- Fairness: On the WinoBias dataset, the "Deviation Ratio" ($\Delta$)—where 0 represents perfect fairness—dropped significantly. For the prompt "CEO," the gender bias score dropped from 0.92 in standard Stable Diffusion to just 0.04.
- Safety: On the I2P benchmark, the method reduced the proportion of inappropriate images to an average of 0.20, outperforming Unified Concept Editing (0.32) and matching Safe Latent Diffusion (0.20).
- Visual Fidelity: Unlike concept-erasing methods that can leave artifacts or "gray out" image sections, this approach maintained high visual quality while successfully shifting the demographics of the generated subjects.
Discussion and Insights
- Disentanglement: Concept whitening moves away from "black box" fixes by creating a continuous semantic space where concepts like "female," "Asian," and "elderly" are mathematically separated for precise control.
- Scalability: The framework is highly extensible. New responsibility concepts (e.g., "smiling" or "wearing glasses") can be added by training small student modules rather than retraining the entire pipeline.
- Limitations: The system currently requires manual hyperparameter tuning to balance the influence of responsibility modules. Additionally, the authors note a dual-use risk, where the same technology could theoretically be used to inject bias if the target set were replaced with irresponsible concepts.
Conclusion
This work provides a robust, interpretable, and scalable solution for responsible T2I generation. By intervening in both the text embedding and diffusion latent spaces, the framework allows for multi-facet control over fairness and safety without compromising the performance of the underlying generative model.
Summary updated