Inference-Only Prompt Projection for Safe Text-to-Image Generation with TV Guarantees

Minhyuk Lee, Wenqian Ye, Myungjoo KangarXiv 2026 · DOI · arXiv

TL;DR

Researchers have formalized the fundamental tension between AI safety and model utility using the lens of Total Variation (TV) distance. They propose an "inference-only" framework that surgically rewrites high-risk prompts into safe versions using a two-stage LLM/VLM cascade, achieving up to a 60% reduction in unsafe content without the "over-alignment" that typically degrades image quality in standard models.

What Problem Does This Paper Address?

Text-to-Image (T2I) models like Stable Diffusion are incredibly powerful but prone to generating "inappropriate" content (violence, hate, etc.). Current solutions usually fall into two camps:

  1. Keyword Blocklists: Easily bypassed by creative prompting.
  2. Model-Level Alignment: Fine-tuning the model or "editing" its weights to remove concepts.

The authors argue that model-level changes are too blunt. Because these updates act globally, they often cause "over-alignment"—where the model's ability to follow benign prompts (like those in the COCO dataset) degrades even when no safety risk is present. This paper introduces the Safety-Prompt Alignment Trade-off (SPAT), a theoretical proof that any reduction in unsafe generations must incur a distributional shift. The goal, then, is to localize that shift only to the prompts that actually need it.

Methods

The core proposal is Inference-Only Prompt Projection. Instead of changing the model, the system changes the input prompt $c$ into a "projected" safe prompt $c^*$ that is semantically close but falls within a safety tolerance $\tau$.

The framework operates via a two-stage cascade:

  • Stage-1: Local Search (LLM): A Large Language Model acts as a "reasoner." It iteratively proposes minimal edits to a high-risk prompt. It uses a surrogate objective (a Lagrangian-style penalty) that balances semantic similarity (measured by sentence embeddings) against a safety score derived from the LLM’s own log-probabilities.
  • Stage-2: Verification (VLM): Once a candidate prompt is selected and an image is generated, a Vision-Language Model (VLM) performs a final check on the actual pixel output. If it fails, the system resamples.

Crucially, this method is near-identity for safe prompts. If a prompt is already benign, the "projection" leaves it effectively untouched, preserving the original model's performance.

Datasets

The authors evaluated their method across three diffusion backbones (SD1.5, SD2.1, and SDXL) using:

  • CoProV2: The primary benchmark for unsafe-prompt evaluation (723 harmful concepts).
  • I2P (Inappropriate Image Prompts): A human-designed set of 4,703 unsafe prompts.
  • Unsafe Diffusion (UD): 932 prompts across 5 categories.
  • COCO-3K: A subset of the COCO dataset used to measure "utility"—ensuring that the safety measures don't break the model's ability to generate normal, high-quality images.

Key Insights and Ideas

  • The TV Lens: By using Total Variation distance, the authors provide a mathematical foundation for why safety and utility are in conflict. It turns the "vibe" of over-alignment into a measurable "distributional shift."
  • Surgical Intervention: Unlike model-editing (which is like performing brain surgery to forget a word), prompt projection is like a polite editor suggesting a rewrite. It keeps the "base generator" fixed, ensuring that the model's creative "knowledge" remains intact.
  • The Safety Knob ($\tau$): The framework introduces a tunable tolerance parameter. A lower $\tau$ makes the system more conservative (higher safety, more prompt changes), while a higher $\tau$ allows for more "borderline" content.
  • Adversarial Robustness: Because the system uses an LLM to "reason" about the prompt rather than just looking for keywords, it is significantly more robust against adversarial attacks designed to bypass simple filters.

Key Results

  • Safety Gains: The method achieved a 16.7–60.0% relative reduction in Inappropriate Percentage (IP) compared to strong baselines like AlignGuard and SLD.
  • Preserving Utility: On the COCO benchmark, the proposed method maintained FID (image quality) and CLIP (text-alignment) scores very close to the unaligned reference model. In contrast, baselines often saw significant drops in these metrics.
  • Efficiency: The two-stage cascade (using the LLM to filter before the expensive VLM check) resulted in a 4.3x–5.1x speedup compared to using a VLM-only verification loop.
  • Cross-Model Success: The approach worked consistently across SD1.5, SD2.1, and SDXL without needing any model-specific retraining.

Limitations and Future Work

  • Semantic Drift: While the authors use "edit-preserving instructions," if a user's intent is fundamentally inseparable from unsafe content, the "projected" prompt will inevitably lose the original meaning.
  • Dependence on Judges: The system's safety is only as good as the LLM and VLM used to score the prompts and images. If the "safety reasoner" has biases or blind spots, the whole pipeline inherits them.
  • Computational Overhead: While faster than some alternatives, performing a local search with an LLM at inference time still adds latency compared to a raw model call.
  • Future Directions: The authors suggest tightening the link between the "ideal" theoretical safety and the "operational" implementation, as well as exploring ways to further reduce semantic drift during rewrites.

Summary updated

Abstract

Text-to-Image (T2I) diffusion models enable high-quality open-ended synthesis, but their real-world deployment demands safeguards that suppress unsafe generations without degrading benign prompt-image alignment. We formalize this tension through a total variation (TV) lens: once the reference conditional distribution is fixed, any nontrivial reduction in unsafe generations necessarily incurs TV deviation from the reference, yielding a principled Safety-Prompt Alignment Trade-off (SPAT). Guided by this view, we propose an inference-only prompt projection framework that selectively intervenes on high-risk prompts via a surrogate objective with verification, mapping them into a tolerance-controlled safe set while leaving benign prompts effectively unchanged, without retraining or fine-tuning the generator. Across four datasets and three diffusion backbones, our approach achieves 16.7-60.0% relative reductions in inappropriate percentage (IP) versus strong model-level alignment baselines, while preserving benign prompt-image alignment on COCO near the unaligned reference.
BibTeX
@misc{lee2026inference,
  author = {Minhyuk Lee and Wenqian Ye and Myungjoo Kang},
  title = {{Inference-Only Prompt Projection for Safe Text-to-Image Generation with TV Guarantees}},
  year = {2026},
  eprint = {2602.00616},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.00616},
}
Figure