When text-to-image (T2I) models are trained on massive, uncurated datasets scraped from the web, they inevitably internalize problematic concepts—ranging from explicit imagery and copyright violations to deep-seated cultural and societal biases. To deploy these models safely, researchers initially relied on surface-level guardrails: blacklisting specific words in user prompts or filtering out unsafe images after generation.
However, recent research demonstrates that these external filters are fundamentally fragile. To build truly robust text-to-image systems, the field is shifting toward internal model interventions—surgically removing harmful concepts from a model's weights and designing context-aware post-processing systems.
The Fragility of Surface-Level Guardrails
Traditional safety filters treat the generative model as a black box, attempting to intercept malicious behavior either at the input stage (text prompts) or the output stage (generated pixels). This approach fails because the mapping between text embeddings and visual latents is highly complex and non-linear.
As a result, safety filters are easily bypassed through implicitly adversarial prompts. These are user inputs that contain entirely benign language but exploit the model's internal correlations to trigger the generation of toxic content. As shown by the authors of AutoPrompt, automated large language models can easily generate human-readable adversarial prompts that completely circumvent standard text filters.
[User Prompt] ---> (Benign Words with Toxic Correlations) ---> [Text Filter: PASS]
|
v
[Harmful Image] <--- (Latent Space Synthesis) <--- [T2I Model Weights]
This vulnerability is not just theoretical. Jessica Quaye et al. demonstrate in Lexically-constrained automated prompt augmentation: A case study using adversarial T2I data that realistic, semantically diverse prompts can consistently trigger safety failures without using blacklisted words. Furthermore, safety mechanisms can be distracted by context. Akhila Yerukola et al. show that when a prompt contains detailed scene descriptions, the model's attention is consumed by rendering the complex environment, causing the safety filter to overlook nuanced harms, such as culturally offensive non-verbal gestures.
This problem is compounded by user behavior in open-source ecosystems. Eddie L. Ungless argues in Typology of Risks of Generative Text-to-Image Models that a significant gap exists between anticipated laboratory risks and observed real-world harms. Empirical studies on open-source model-sharing platforms by Yiluo Wei et al. and Maria-Teresa De Rosa Palmini et al. reveal that users actively migrate to open-source models specifically to bypass commercial safety filters, frequently subverting model-level labels through clever prompting.
Deep Defense via Concept Erasure
Because surface-level filters can be bypassed, a more permanent solution is to modify the model so that it is physically incapable of representing or constructing harmful concepts. This strategy is known as concept erasure.
The intuition behind concept erasure is that specific semantic concepts are often localized within distinct sub-networks or parameters of a deep model. In ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruning, Ruchika Chavhan et al. demonstrate that a tiny fraction of a diffusion model's weights (~0.12%) act as "skilled neurons" responsible for rendering specific unwanted concepts like nudity or copyrighted styles. By zeroing out these specific neurons, the model completely forgets how to generate the target concept, creating a defense that is highly robust against adversarial jailbreaks.
[Standard Model Weights] ---> Contains "Skilled Neurons" for Nudity/Copyright
|
v (Surgical Pruning / Trajectory Matching)
[Immunized Model Weights] ---> Concept Erased; Adversarial Prompts Fail to Render Target
Other approaches modify the model's generation trajectories rather than just pruning weights. Jonas Henry Grebe et al. introduce GEM: Geometric Erasure by Contrastive Velocity Matching in Rectified Flows, which uses teacher-guided velocity matching to alter the generation paths of rectified flow models, steering them away from unsafe visual spaces. To ensure that erasing an unsafe concept does not inadvertently degrade benign content, Lingyun Zhang et al. propose Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization. This framework uses a localized cross-attention module to swap out sensitive concepts (e.g., replacing a specific face or explicit object) while leaving the rest of the image's style and composition completely intact.
Defenders must also anticipate adversarial adaptation. Even if a base model is safely released, malicious actors can use fine-tuning techniques like DreamBooth or Textual Inversion to re-introduce harmful concepts. To prevent this, Amber Yijia Zheng et al. propose a method called MIMA in Multi-concept Model Immunization through Differentiable Model Merging. MIMA uses a bi-level optimization framework to "immunize" the model's pre-trained weights, making them mathematically resistant to being fine-tuned on multiple harmful concepts simultaneously.
Balancing Safety and Utility via Post-Hoc Editing
While concept erasure provides a robust defense, aggressive weight modification introduces a new challenge: over-refusal. When safety boundaries are drawn too broadly, models often fail to generate benign historical, educational, or artistic imagery. As Viacheslav Vasilev et al. demonstrate in RusCode: Russian Cultural Code Benchmark for Text-to-Image Generation, aggressive safety filters frequently suppress legitimate cultural and historical depictions because they mistake benign local identities for sensitive content.
To resolve this tension, researchers are moving away from binary "block-or-allow" paradigms toward intelligent, context-aware remediation.
[User Prompt] ---> [T2I Model] ---> [Raw Image (Contains Violation)]
|
v
[Multimodal LLM]
|---> Reasons about the violation
|---> Edits ONLY the unsafe component
v
[Safe, Preserved Image]
In SafeEditor: Unified MLLM for Efficient Post-hoc T2I Safety Editing, Xiaoru Feng et al. introduce a post-hoc editing framework powered by a Multimodal Large Language Model (MLLM). Instead of blocking a prompt or deleting a violating image entirely, the MLLM analyzes the generated output, reasons about the specific safety violation, and iteratively edits only the unsafe components. By separating the creative generation process from the safety enforcement process, this paradigm allows text-to-image models to maintain maximum utility and creative freedom while ensuring the final visual output remains safe.
Key Foundational Work
Typology of Risks of Generative Text-to-Image Models (2023) — contributed 3 concept elements to this direction.
Ensuring Visual Commonsense Morality for Text-to-Image Generation (2023) — contributed 1 concept elements to this direction.
ARTIST: A Transformer-based Chinese Text-to-Image Synthesizer Digesting Linguistic and World Knowledge (2022) — contributed 1 concept elements to this direction.