Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control

Basim Azam, Naveed Akhtar

Abstract

Ethical issues around text-to-image (T2I) models demand a comprehensive control over the generative content. Existing techniques addressing these issues for responsible T2I models aim for the generated content to be fair and safe (non-violent/explicit). However, these methods remain bounded to handling the facets of responsibility concepts individually, while also lacking in interpretability. Moreover, they often require alteration to the original model, which compromises the model performance. In this work, we propose a unique technique to enable responsible T2I generation by simultaneously accounting for an extensive range of concepts for fair and safe content generation in a scalable manner. The key idea is to distill the target T2I pipeline with an external plug-and-play mechanism that learns an interpretable composite responsible space for the desired concepts, conditioned on the target T2I pipeline. We use knowledge distillation and concept whitening to enable this. At inference, the learned space is utilized to modulate the generative content. A typical T2I pipeline presents two plug-in points for our approach, namely; the text embedding space and the diffusion model latent space. We develop modules for both points and show the effectiveness of our approach with a range of strong results.

Introduction: The Challenge of Responsible Generation

Large-scale text-to-image (T2I) models, such as Stable Diffusion, are typically trained on massive, uncurated web-scale datasets. While powerful, these models often inherit and amplify societal biases—for instance, defaulting to specific demographics for certain professions like "CEO" or "doctor." Furthermore, they can inadvertently generate unsafe or inappropriate content.

Existing mitigation strategies generally fall into three categories: prompt filtering, model fine-tuning, or post-hoc moderation. However, these approaches face significant hurdles. Prompt filtering is easily bypassed; fine-tuning is computationally expensive and often degrades overall image quality; and most methods struggle to manage multiple ethical concepts simultaneously without causing "concept interference." There is a critical need for a scalable, interpretable, and non-invasive method to enforce responsibility in T2I generation.

Methodology: A Dual-Space Approach

This paper introduces a "plug-and-play" framework that intervenes at two critical stages of the T2I pipeline without requiring retraining of the base model:

  1. RICE (Responsible Interpretable CLIP Embedding): This module operates within the text encoder space. It employs knowledge distillation to learn a "responsible" version of text embeddings.
  2. RIIDL (Responsible Interpretable Intermediate Diffusion Latents): This module targets the diffusion model's U-Net. It nudges the noisy latents during the generation process to align with desired responsibility concepts.

Concept Whitening for Interpretability

A core innovation is the use of concept whitening. This technique decorrelates semantic concepts (e.g., age, gender, race) within the learned space. By disentangling these attributes, the system allows for precise, independent control. For example, a user can adjust representation for gender without unintentionally affecting the racial distribution of the output. These modules act as additive layers that can be toggled or weighted based on specific requirements.

Evaluation and Benchmarks

The framework was evaluated using two primary benchmarks:

  • WinoBias: Used to measure occupational bias across 35 professions. The primary metric is the deviation ratio ($\Delta$), which compares the model's output distribution against a fair baseline.
  • I2P (Inappropriate-to-Prompt): A safety benchmark containing prompts designed to trigger inappropriate content, including violence, hate speech, and sexual imagery.

Key Results

The proposed dual-space method demonstrates significant improvements in both fairness and safety:

  • Effective Debiasing: On the WinoBias benchmark, the method consistently outperformed state-of-the-art competitors. For example, the bias in the "CEO" prompt was reduced from a deviation ratio of 0.92 in standard Stable Diffusion to just 0.04.
  • Safety Enhancements: In the I2P benchmark, the system significantly reduced inappropriate outputs, performing on par with or better than dedicated safety models like Safe Latent Diffusion (SLD).
Diverse representation for the prompt 'doctor' achieved through multi-facet control.
  • Multi-Facet Control: Qualitative analysis shows the model's ability to smoothly transition through various combinations of age, race, and gender attributes while preserving the core identity of the prompt.
Suppression of inappropriate content using the proposed safety modulation.

Discussion and Future Work

While the framework offers a robust solution for responsible generation, it has some limitations. The system's performance is sensitive to hyperparameters ($\lambda$ and $\gamma$) used to balance responsibility with prompt adherence, currently requiring manual tuning. Additionally, while scalable, managing a very high number of simultaneous concepts could lead to signal noise.

The authors also note a "dual-use" risk: the same mechanism designed for responsibility could theoretically be used to inject harmful concepts if misused. Future research will focus on automating hyperparameter selection and exploring more sophisticated management of high-dimensional concept spaces to prevent interference.

Summary updated

Abstract

Ethical issues around text-to-image (T2I) models demand a comprehensive control over the generative content. Existing techniques addressing these issues for responsible T2I models aim for the generated content to be fair and safe (non-violent/explicit). However, these methods remain bounded to handling the facets of responsibility concepts individually, while also lacking in interpretability. Moreover, they often require alteration to the original model, which compromises the model performance. In this work, we propose a unique technique to enable responsible T2I generation by simultaneously accounting for an extensive range of concepts for fair and safe content generation in a scalable manner. The key idea is to distill the target T2I pipeline with an external plug-and-play mechanism that learns an interpretable composite responsible space for the desired concepts, conditioned on the target T2I pipeline. We use knowledge distillation and concept whitening to enable this. At inference, the learned space is utilized to modulate the generative content. A typical T2I pipeline presents two plug-in points for our approach, namely; the text embedding space and the diffusion model latent space. We develop modules for both points and show the effectiveness of our approach with a range of strong results. Our code and models will be made public after paper acceptance.
BibTeX
@inproceedings{azam2025plug,
  author = {Basim Azam and Naveed Akhtar},
  title = {{Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=8YVHooRI89},
}
Figure