TL;DR
PreviewDiff is a training-free method that transforms diffusion sampling from a blind, "open-loop" process into a guided, "closed-loop" search. By decoding intermediate latent previews and using a multimodal LLM to provide natural-language critiques and prompt edits, the system can detect and repair compositional errors—like wrong object counts or spatial mismatches—before the generation is finished.
What Problem Does This Paper Address?
While modern diffusion models produce stunningly realistic images and videos, they are notoriously brittle when it comes to following complex prompts. They often fail at "compositional" tasks: they might get the number of objects wrong, swap colors between subjects (attribute binding errors), or misinterpret spatial relationships (e.g., putting a sheep under an umbrella instead of the other way around).
The standard way to fix this is "Best-of-N" sampling—generating many versions and picking the best one at the end. However, this is inefficient. If a generation starts failing early in the denoising process, Best-of-N simply finishes the "bad" trajectory and discards it. The authors argue that we should instead use that computation to intervene early, diagnosing and steering promising but flawed trajectories while they are still editable.
Methods
PreviewDiff introduces a multimodal critic-guided search over intermediate latents. The process works as follows:
- Intermediate Previews: At specific checkpoints during the denoising process, the system decodes the current noisy latent into a "clean" estimate preview.
- Multimodal Critique: A multimodal judge (like Gemini 3.1 Flash) inspects the preview against the original prompt. It provides a score and, crucially, natural-language "fix notes" (e.g., "Separate the pig from the backpack").
- Semantic Branching: Instead of just continuing, the system creates multiple "branches." It appends the critic's fix notes to the prompt and performs a local restart—slightly re-noising the latent to give the model room to incorporate the new instructions.
- Beam-Pruned Search: The system uses a tree search strategy to evaluate these branches. It rolls them forward to the next checkpoint, scores them again, and keeps only the most promising candidates (the "beam").
This approach is entirely training-free, meaning it can be wrapped around any existing pre-trained diffusion model without fine-tuning.
Datasets
The authors evaluated PreviewDiff across several rigorous benchmarks for both images and videos:
- Images: GenEval2 (the primary scaling benchmark focusing on object-centric constraints), T2I-CompBench++, and T2I-CoReBench.
- Videos: NarrLV (a new benchmark for narrative and temporal structure), T2V-CompBench, and VBench 2.0.
These datasets were chosen specifically because they stress the compositional and temporal weaknesses that standard metrics like FID often miss.
Key Insights and Ideas
- Feedback as an Active Controller: The most significant insight is that multimodal feedback is more effective when used as an active controller during generation rather than just a final verifier.
- Language is a Powerful Steering Tool: Unlike previous search methods that used simple scalar rewards (a single number), PreviewDiff uses natural language. This allows the critic to give specific, interpretable directions that the diffusion model can actually follow.
- Early Intervention Wins: Ablations showed that intervening at early denoising steps provides the largest gains. Even if the early preview is noisy, it contains enough semantic structure for a critic to identify if the layout is going off-track.
- Search Width > Depth: Increasing the "search width" (trying more parallel branches from a single point) was more effective than increasing the "depth" (intervening many times in a single trajectory).
Key Results
PreviewDiff consistently outperformed budget-matched Best-of-N baselines (where the same amount of total computation/LLM calls were used) across all models tested, including SDXL, FLUX.2, and LTX-Video.
- Image Generation (SDXL): PreviewDiff improved the mean Gemini score from 2.17 to 2.86 as compute increased, significantly higher than the 2.71 achieved by Best-of-N.
- Video Generation (LTX-Video): The method improved scores from 1.99 to 2.39, compared to 2.13 for Best-of-N.
- Human Preference: In human evaluations, PreviewDiff was strongly preferred for content accuracy, beating both Best-of-N and other search baselines like EvoSearch.
Limitations and Future Work
Despite the improvements, PreviewDiff is not a silver bullet. The authors noted several remaining challenges:
- Exact Numeracy: While it helps with counting, the model can still struggle with precise high numbers (e.g., "exactly 12 turtles").
- Long-Horizon Temporal Logic: In video, it can still miss complex sequences of actions (e.g., the specific steps of lighting and then blowing out a candle).
- Computational Cost: Every branch expansion requires a visual decode and an MLLM call, which adds significant latency compared to a single feed-forward pass.
Future work could involve optimizing the "local restart" mechanism or developing more specialized critics that are better at temporal reasoning for video.
Summary updated