Warm Starts Accelerate Generative Modelling

Jonas Scholz, Richard E. Turner

Introduction

Generative models, particularly diffusion and flow-matching frameworks, have become the standard for high-fidelity data synthesis. However, their inference speed remains a significant bottleneck. Traditionally, these models begin the generative process from a state of pure white noise—a standard Gaussian distribution $\mathcal{N}(0, I)$—and iteratively refine this noise into a clean sample. This "uninformed" starting point often necessitates dozens or even hundreds of Number of Function Evaluations (NFE) to produce high-quality results.

In conditional generation tasks such as image inpainting or weather forecasting, we often possess substantial prior context. Starting from completely uninformed noise ignores this available information, forcing the model to reconstruct basic structures from scratch in every sample. This paper introduces Warm-Start Diffusion (WSD), a method that leverages conditional context to provide a "warm start," significantly reducing the computational distance to the target data distribution.

Methodology: Warm-Start Diffusion

WSD accelerates the generative process by replacing the standard noise prior with an informed prior predicted by a lightweight auxiliary model.

Overview of Warm-Start Diffusion (WSD) compared to standard diffusion. Panel (a) shows standard diffusion from pure noise. Panel (b) illustrates the informed prior in data space. Panel (c) shows the normalization trick that allows WSD to use standard solvers.

The framework consists of three core components:

  1. The Warm-Start Model ($h_\phi$): A small, deterministic neural network (e.g., a lightweight U-Net) that processes the context $C$ to predict the first two moments of the conditional data distribution: a mean $\hat{\mu}_C$ and a marginal standard deviation $\hat{\sigma}_C$.
  2. The Informed Prior: Instead of $\mathcal{N}(0, I)$, the process begins at $\mathcal{N}(\hat{\mu}_C, \text{diag}(\hat{\sigma}^2_C))$. This starting point is physically much closer to the final data manifold.
  3. The Conditional Normalization Trick: To maintain compatibility with existing diffusion solvers without requiring complex re-derivations, the authors apply a per-instance transformation: $X'_t = (X_t - \hat{\mu}_C) / \hat{\sigma}_C$. This maps the informed prior back into a standard Gaussian space ($X'_T \sim \mathcal{N}(0, I)$), allowing standard algorithms to operate in this "normalized" space.
  4. Warmth Blending: To ensure the model remains robust across different NFE regimes, a "warmth" parameter $w$ is used during training to blend the warm-start objective with standard diffusion.

Experimental Evaluation

The authors evaluated WSD across three diverse domains:

  • CelebA (64x64): Face image inpainting (95% pixel removal).
  • CIFAR-10 (32x32): Natural image inpainting (90% pixel removal) and ablation studies.
  • ERA5 Wind Forecasting: Predicting future wind fields based on historical meteorological states.

Key Results

WSD demonstrates remarkable efficiency gains. In image inpainting tasks, it generates realistic samples in just 4–6 NFE, whereas standard flow-matching typically requires over 20 NFE to achieve comparable quality. Performance tends to saturate early, around 10–12 NFE, reaching peak FID scores much faster than baselines. In the ERA5 weather forecasting task, WSD achieved a 75% reduction in compute while maintaining the ability to generate diverse, physically plausible ensemble forecasts.

Discussion and Insights

A critical insight from the authors' ablation studies is the importance of variance. Simply subtracting the mean (Mean-only) is insufficient; predicting per-pixel standard deviation is essential for the efficiency gains observed in low-NFE regimes.

Furthermore, WSD is orthogonal to improvements in ODE solvers. While most research focuses on taking larger steps (increasing step size), WSD reduces the total distance the solver must travel. Combining WSD with advanced solvers like DPM-Solver yields a compound acceleration effect.

Limitations

The current implementation of WSD assumes a unimodal Gaussian posterior. While effective for "strong" conditioning (like inpainting), it may struggle with "weak" conditioning (like text-to-image) where the target distribution is highly multimodal. Additionally, the current approach requires training a task-specific warm-start model. Future work may explore general-purpose warm-start models or more complex posterior approximations, such as low-rank correlation matrices.

Summary updated

BibTeX
@misc{scholz2025warm,
  author = {Jonas Scholz and Richard E. Turner},
  title = {{Warm Starts Accelerate Generative Modelling}},
  year = {2025},
  eprint = {2507.09212},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.09212},
}
Figure