Haste Makes Waste? Towards Safe Few-Step Text-to-Image Models

Lior Ziv, Mahmood SharifarXiv 2026 · arXiv

TL;DR

Distilling text-to-image models for speed (1–4 steps) creates a "safety gap": these models generate unsafe content more frequently than their slower "teacher" models and resist standard concept-erasure techniques. This paper introduces DICE (Distillation-Integrated Concept Erasure), a method tailored for few-step models that significantly reduces unsafe generation while maintaining high image quality.

What Problem Does This Paper Address?

Modern text-to-image (T2I) models like FLUX and Stable Diffusion 3.5 are often deployed as "few-step" distilled variants to save on compute and latency. While these models are fast, their safety properties haven't been systematically audited.

The authors identify two major issues:

  1. The Safety-Speed Trade-off: As you reduce the number of inference steps, the rate of unsafe content (like nudity or violence) consistently increases.
  2. Erasure Resistance: Standard methods for "erasing" concepts (e.g., fine-tuning a model to stop generating nudity) work well on full-step models but fail miserably on their few-step students.

The core challenge is that the distillation process—which compresses 50 steps of reasoning into 4—changes the model's internal geometry and how it handles text guidance, making post-hoc safety edits fragile and ineffective.

Methods

To solve these issues, the authors propose DICE, a fine-tuning method using Low-Rank Adaptation (LoRA). DICE differs from standard erasure methods in three key ways:

  • Straight-Path Latent Construction: Standard erasure methods sample training data "on-policy" (using the model's own noisy generation path). Because few-step models are highly sensitive to noise, DICE instead uses "off-policy" latents sampled along a perfectly straight line between Gaussian noise and a clean concept image. This keeps the erasure updates aligned with the simplified trajectory the distilled model was trained to follow.
  • Self-Distilled Weight Preservation: To prevent the model's general image quality from degrading during erasure, DICE uses a "generation-preservation loss." Instead of trying to match a generic training target, it forces the edited model to stay close to the original distilled student's outputs for benign prompts.
  • Attention-Map Suppression: Beyond just changing the final image output, DICE directly penalizes the model's internal attention to "unsafe" text tokens (e.g., "nude"), dampening the concept at its source within the transformer blocks.

Datasets

The researchers evaluated their work using several key benchmarks:

  • NSFW-56k / NSFW-1k: A dataset of highly explicit prompts used to measure nudity generation rates.
  • I2P (Image-to-Prompt): A standard benchmark for inappropriate content, though noted by the authors as being less explicit than NSFW-56k.
  • T2I-RiskyPrompt: Used specifically to evaluate the erasure of violent content.
  • LAION-Pop: Used as a source of "clean" images for the off-policy training and sensitivity analysis.
  • COCO-30k: Used to evaluate general image quality via FID and CLIP-similarity scores.
  • HPSv2: A metric based on human preference to ensure the "benign" generation quality remains high.

Key Insights and Ideas

  • Haste Really Does Make Waste: Reducing inference steps isn't safety-neutral. For example, lowering the step count can raise the mean CLIP similarity to "nudity" by up to 9% in full-step models. CLIP similarity to "nudity" increases as step count decreases.
CLIP similarity to "nudity" increases as step count decreases.
  • Geometric Fragility: Aggressively distilled models are much more sensitive to perturbations. Their "Lipschitz constant" (a measure of how much the output changes relative to a small change in input or weights) is 2x to 5x higher than their teachers. This makes them "brittle" during fine-tuning. Few-step models (FLUX Schnell) show much higher sensitivity than their teachers (FLUX Dev).
Figure 4. Empirical local sensitivity of the velocity field with respect to latent z perturbations in FLUX Dev and FLUX Schnell, plotted by noise level σt. Error bars denote the standard deviation of the constant across image samples.
  • Guidance Bake-in Distorts Erasure: In many few-step models, Classifier-Free Guidance (CFG) is "baked" into the weights. This distorts the "concept direction" that erasure methods rely on, rotating it away from the teacher's direction and inflating its magnitude, leading to failed edits.

Key Results

DICE outperformed all existing erasure methods (like ESD and EraseAnything) across multiple model families, including FLUX Schnell and SD3.5 Turbo.

  • Safety Improvement: On SD3.5 Turbo, DICE reduced the NudeNet unsafe rate from 0.323 (undefended) to 0.032. The best prior method (ESD) only reached 0.150.
  • Quality Preservation: Despite the aggressive safety filtering, the HPSv2 quality score only dropped slightly (e.g., from 0.287 to 0.283 on SD3.5 Turbo), meaning the model still produces high-quality images for safe prompts.
  • Adversarial Robustness: In tests against automated "jailbreak" attacks like Ring-A-Bell, DICE remained the most robust defense, though the authors admit that determined adversaries can still find bypasses. Comparison of NudeNet rates across different integration strategies.
Figure 3. NudeNet rates on NSFW-1k for EraseAnything using post-distillation erasure, pre-distillation erasure, and Teacher-LoRA transfer erasure application methods. Grouped by model family, FLUX SCFM is tested as FLUX Dev’s student and SD3.5 Turbo as SD3.5 Large’s student. For SD3.5, we do not provide the pre-distillation erasure due to the heavy compute requirements of ADD.

Limitations and Future Work

  • Not a Perfect Shield: While DICE raises the bar, it is not "bulletproof" against sophisticated white-box adversarial attacks.
  • Scope of Harms: The study focused primarily on nudity and violence. The authors note that other harms—like deepfakes, political misinformation, and training data leakage—need similar dedicated audits in the few-step regime.
  • Evaluation Bottlenecks: The work relies on automated safety classifiers (like NudeNet), which may not always align perfectly with human judgment. Developing safe protocols for human evaluation of explicit content remains an open challenge.

Summary updated

Abstract

While few-step distilled text-to-image models have gained traction due to their fast generation times, the speed-up implications on their unsafe generation rates remain unknown. We uncover an inherent tension between speed and safety: few-step models are often more unsafe than their teachers, and reducing the number of inference steps consistently raises unsafe generation rates. We further find that acceleration makes unsafe concepts harder to erase in few-step models, with existing concept-erasure methods markedly less effective than in teachers. We offer a mechanistic diagnosis for why erasure methods fail on few-step models, finding that the fragility of few-step models' generation process and the way generation guidance is usually distilled into those models may hinder established concept-erasure methods. To address the speed-safety trade-off, we propose DICE (Distillation-Integrated Concept Erasure), a post-distillation erasure method tailored for few-step models. DICE aligns both its latent construction and its generation-preservation loss with the distillation method that produced the model, keeping erasure faithful to few-step models. DICE demonstrates strong results in our evaluation. For example, on FLUX Schnell, DICE reduces unsafe-generation rate to 0.046 (vs. at least 0.161 for prior erasure methods we test), while mostly preserving benign generation quality.
BibTeX
@misc{ziv2026haste,
  author = {Lior Ziv and Mahmood Sharif},
  title = {{Haste Makes Waste? Towards Safe Few-Step Text-to-Image Models}},
  year = {2026},
  eprint = {2610.36665},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36665},
}
Figure