PASTA: Prompt-Aware Annealed Diffusion Sampling

Ren Ajiki, Ryugo Morita, Stanislav Frolov, Brian Bernhard Moser, Ko Watanabe, Riku Takahashi, Andreas Dengel, Jinjia ZhouarXiv 2026 · arXiv

TL;DR

PASTA (Prompt-Aware Shared-Latent Training-Free Annealed Diffusion Sampling) is a new training-free method that allows users to reserve "clean" regions in AI-generated images for text, logos, or UI elements. By fusing different prompt-based guidance residuals on a shared latent trajectory and using a time-dependent annealing schedule, PASTA keeps reserved areas free of foreground objects while maintaining a coherent, natural background.

What Problem Does This Paper Address?

While modern diffusion models like SDXL generate stunning images, they are notoriously difficult to control for professional design workflows. Designers often need "negative space"—specific areas left intentionally clear for headlines or branding.

Existing "layout-aware" methods fall into two camps, both with significant drawbacks:

  1. Attention-based methods (e.g., TextCenGen): These try to "push" object attention away from a region. They often fail when the reserved area is large, leading to "semantic leakage" where parts of an object still bleed into the forbidden zone.
  2. Noise-level interventions (e.g., SAWNA): These suppress object generation by injecting non-reactive noise. While they keep the area clear, they often create "dead zones"—flat, monochrome, or visually inconsistent patches that look like a bad Photoshop job.

PASTA aims to solve this "fidelity vs. consistency" trade-off: keeping the reserved region strictly empty of foreground objects without sacrificing the visual flow of the background.

A comparison showing how PASTA maintains background coherence compared to existing methods.

Methods

PASTA is a training-free sampling framework, meaning it works on top of existing pre-trained models like SDXL without needing extra fine-tuning. Its architecture relies on three core components:

  • Prompt Decomposition: The system uses an LLM (like ChatGPT) to split a user's prompt into a "foreground phrase" (the object to be excluded) and a "context phrase" (the background/style). For example, "An elephant in a forest" becomes "An elephant" (foreground) and "The forest" (context).
  • Shared-Latent Residual Fusion: At each denoising step, PASTA calculates three different noise predictions—Full Prompt, Context Prompt, and Negative Prompt—all based on the exact same latent state. By using a shared latent, the different guidance directions stay perfectly aligned spatially, preventing "seams" in the final image.
  • Annealed Blending Schedule: This is the "secret sauce." In the early stages of generation (when the layout is being decided), PASTA applies heavy "context-only" guidance to the reserved region to prevent objects from even starting to form. As the process continues, it gradually eases off, allowing the global "full prompt" guidance to refine the background and ensure the lighting and textures match across the entire image.
Fig. 2: Overview of PASTA. Given a full prompt Pfull and a reserved-region mask M, PASTA computes guidance residuals from shared-latent noise predictions un- der full-prompt, context-prompt, and negative-prompt conditions. The residuals are spatially fused to suppress foreground-object guidance inside the reserved region, while an annealed schedule controls the context-prompt contribution in the object-allowed region.

Datasets

The authors evaluated PASTA using two standard benchmarks for space-aware generation:

  • P2P (Prompt-to-Prompt): A dataset of paired prompts and spatial masks.
  • SynPrompt: A dataset specifically designed for text-centric background adaptation.

Crucially, the authors introduced a new "large-mask" evaluation protocol. While previous benchmarks used tiny masks (averaging 3.8% of the image), PASTA was tested against masks covering ~30% of the image to simulate realistic design constraints.

Key Insights and Ideas

  • The "Shared Latent" Advantage: Most multi-prompt methods suffer from "trajectory drift," where different prompts pull the image in different stylistic directions. By forcing all predictions to stem from the same latent, PASTA ensures that the "forest" inside the reserved box looks exactly like the "forest" outside it.
  • Timing is Everything: The paper highlights that semantic leakage happens early. If an object's "seed" is planted in the first few steps, it's almost impossible to remove later. The annealing schedule targets these critical early steps for maximum suppression.
  • Context over Silence: Unlike previous methods that just try to "silence" the reserved region, PASTA actively guides it with the context prompt. This ensures the region isn't just empty, but is filled with relevant background details (like sky or grass) that match the rest of the scene.

Key Results

PASTA was compared against SDXL (baseline), TextCenGen, and SAWNA. The results were measured using Intrusion Rate (how many foreground pixels leaked into the mask) and perceptual metrics like CLIP Score and HPSv2.

  • Superior Suppression: In the large-mask setting on the P2P dataset, PASTA achieved an intrusion rate of 0.038, significantly lower than TextCenGen (0.222) and the SDXL baseline (0.223).
  • Visual Fidelity: While SAWNA achieved a lower intrusion rate in some cases, it suffered from poor visual quality. PASTA maintained much higher HPSv2 scores, indicating that the images looked more "natural" to human observers.
  • Ablation Success: Qualitative tests showed that without the annealing schedule, the model either failed to keep the region clean or produced "shrunken" foreground objects that looked unnatural.
Ablation study showing the effect of the annealing schedule on object size and region cleanliness.

Limitations and Future Work

The authors identify a "breaking point" for the model: extreme spatial constraints. If a user tries to reserve more than 80% of the image, or leaves only a very narrow sliver for the foreground object, the model struggles to produce a plausible image. This is likely because the underlying diffusion model (SDXL) hasn't seen such extreme layouts during its original training.

Future work may involve "data-driven" or adaptive training approaches to help models understand these highly constrained, designer-centric layouts better.

Summary updated

Abstract

While Diffusion models produce high-quality images, real-world visual design requires intentionally preserving clean regions for headlines, logos, or UI elements. Existing layout-aware text-to-image methods struggle to strictly preserve reserved regions, often introducing unwanted content that complicates downstream editing. Existing approaches either manipulate attention to displace content or suppress conditioning via noise-level interventions. However, these methods sacrifice visual coherence, producing flat regions or noticeable style inconsistencies between reserved and non-reserved areas. To address this limitation, we propose a Prompt-Aware Shared-Latent Training-Free Annealed Diffusion Sampling (PASTA), a training-free framework that enforces reserved-region constraints by selectively applying semantic guidance to object-allowed regions during sampling. PASTA operates on a shared latent trajectory, where full-prompt and context-prompt guidance are fused through mask-guided sampling. An annealed schedule suppresses semantic leakage during early denoising stages while gradually restoring visual consistency in later stages. Extensive experiments demonstrate that PASTA improves reserved-region fidelity while preserving overall image realism and perceptual consistency.
BibTeX
@misc{ajiki2026pasta,
  author = {Ren Ajiki and Ryugo Morita and Stanislav Frolov and Brian Bernhard Moser and Ko Watanabe and Riku Takahashi and Andreas Dengel and Jinjia Zhou},
  title = {{PASTA: Prompt-Aware Annealed Diffusion Sampling}},
  year = {2026},
  eprint = {2610.32390},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32390},
}
Figure