TL;DR
Distilling text-to-image models for speed (1–4 steps) creates a "safety gap": these models generate unsafe content more frequently than their slower "teacher" models and resist standard concept-erasure techniques. This paper introduces DICE (Distillation-Integrated Concept Erasure), a method tailored for few-step models that significantly reduces unsafe generation while maintaining high image quality.
What Problem Does This Paper Address?
Modern text-to-image (T2I) models like FLUX and Stable Diffusion 3.5 are often deployed as "few-step" distilled variants to save on compute and latency. While these models are fast, their safety properties haven't been systematically audited.
The authors identify two major issues:
- The Safety-Speed Trade-off: As you reduce the number of inference steps, the rate of unsafe content (like nudity or violence) consistently increases.
- Erasure Resistance: Standard methods for "erasing" concepts (e.g., fine-tuning a model to stop generating nudity) work well on full-step models but fail miserably on their few-step students.
The core challenge is that the distillation process—which compresses 50 steps of reasoning into 4—changes the model's internal geometry and how it handles text guidance, making post-hoc safety edits fragile and ineffective.
Methods
To solve these issues, the authors propose DICE, a fine-tuning method using Low-Rank Adaptation (LoRA). DICE differs from standard erasure methods in three key ways:
- Straight-Path Latent Construction: Standard erasure methods sample training data "on-policy" (using the model's own noisy generation path). Because few-step models are highly sensitive to noise, DICE instead uses "off-policy" latents sampled along a perfectly straight line between Gaussian noise and a clean concept image. This keeps the erasure updates aligned with the simplified trajectory the distilled model was trained to follow.
- Self-Distilled Weight Preservation: To prevent the model's general image quality from degrading during erasure, DICE uses a "generation-preservation loss." Instead of trying to match a generic training target, it forces the edited model to stay close to the original distilled student's outputs for benign prompts.
- Attention-Map Suppression: Beyond just changing the final image output, DICE directly penalizes the model's internal attention to "unsafe" text tokens (e.g., "nude"), dampening the concept at its source within the transformer blocks.
Datasets
The researchers evaluated their work using several key benchmarks:
- NSFW-56k / NSFW-1k: A dataset of highly explicit prompts used to measure nudity generation rates.
- I2P (Image-to-Prompt): A standard benchmark for inappropriate content, though noted by the authors as being less explicit than NSFW-56k.
- T2I-RiskyPrompt: Used specifically to evaluate the erasure of violent content.
- LAION-Pop: Used as a source of "clean" images for the off-policy training and sensitivity analysis.
- COCO-30k: Used to evaluate general image quality via FID and CLIP-similarity scores.
- HPSv2: A metric based on human preference to ensure the "benign" generation quality remains high.
Key Insights and Ideas
- Haste Really Does Make Waste: Reducing inference steps isn't safety-neutral. For example, lowering the step count can raise the mean CLIP similarity to "nudity" by up to 9% in full-step models. CLIP similarity to "nudity" increases as step count decreases.
- Geometric Fragility: Aggressively distilled models are much more sensitive to perturbations. Their "Lipschitz constant" (a measure of how much the output changes relative to a small change in input or weights) is 2x to 5x higher than their teachers. This makes them "brittle" during fine-tuning. Few-step models (FLUX Schnell) show much higher sensitivity than their teachers (FLUX Dev).
- Guidance Bake-in Distorts Erasure: In many few-step models, Classifier-Free Guidance (CFG) is "baked" into the weights. This distorts the "concept direction" that erasure methods rely on, rotating it away from the teacher's direction and inflating its magnitude, leading to failed edits.
Key Results
DICE outperformed all existing erasure methods (like ESD and EraseAnything) across multiple model families, including FLUX Schnell and SD3.5 Turbo.
- Safety Improvement: On SD3.5 Turbo, DICE reduced the NudeNet unsafe rate from 0.323 (undefended) to 0.032. The best prior method (ESD) only reached 0.150.
- Quality Preservation: Despite the aggressive safety filtering, the HPSv2 quality score only dropped slightly (e.g., from 0.287 to 0.283 on SD3.5 Turbo), meaning the model still produces high-quality images for safe prompts.
- Adversarial Robustness: In tests against automated "jailbreak" attacks like Ring-A-Bell, DICE remained the most robust defense, though the authors admit that determined adversaries can still find bypasses. Comparison of NudeNet rates across different integration strategies.
Limitations and Future Work
- Not a Perfect Shield: While DICE raises the bar, it is not "bulletproof" against sophisticated white-box adversarial attacks.
- Scope of Harms: The study focused primarily on nudity and violence. The authors note that other harms—like deepfakes, political misinformation, and training data leakage—need similar dedicated audits in the few-step regime.
- Evaluation Bottlenecks: The work relies on automated safety classifiers (like NudeNet), which may not always align perfectly with human judgment. Developing safe protocols for human evaluation of explicit content remains an open challenge.
Summary updated