The role of injected noise in diffusion model generalization

Vincent Emmerling, Felix Günther, David KappelarXiv 2026 · arXiv

TL;DR

Diffusion models trained on different data can generate nearly identical images if they share the same noise inputs. This paper reveals that the sequence of noise injected at each step of the sampling process, rather than the initial starting noise, is the primary driver of this "strong generalization," with its impact peaking during a critical middle phase called the "speciation regime."

What Problem Does This Paper Address?

Recent research has shown a surprising phenomenon: two different diffusion models (DMs), even those trained on non-overlapping subsets of data, will produce highly similar images when given the same random noise. This suggests that DMs learn a universal, generalized data distribution rather than just memorizing their training sets.

However, the sampling process in DMs involves two distinct sources of randomness:

  1. Starting Noise ($x_0$): The initial Gaussian noise vector.
  2. Injected Noise Sequence ($z_t$): The independent noise samples added at every single denoising step.

Until now, it wasn't clear which of these components—or which specific time steps during sampling—contributed most to the final image's identity and the model's ability to generalize. The authors set out to disentangle these effects to understand the "where and when" of diffusion model consistency.

Methods

The authors employed a standard diffusion sampling framework based on the Euler-Maruyama approximation of a Stochastic Differential Equation (SDE). Their core experimental innovation was to explicitly control and swap the $x_0$ and $z_t$ components.

  • Consistency Analysis: They compared images generated by different models (Model A vs. Model B) and different starting conditions. They measured similarity using cosine similarity between the final images and intermediate states.
  • Intermediate State Inspection: They tracked both the noisy state $x_t$ and the "denoised state" $\hat{x}_t$ ( the model's best guess of the clean image at step $t$).
  • Regime Ablation: Following previous work that divides diffusion into three phases—Regime I (Brownian motion), Regime II (Speciation), and Regime III (Collapse)—the authors replaced 20% segments of the injected noise sequence with different noise to see which phase was most sensitive to change.

Datasets

  • CelebA: A large-scale face attributes dataset. The authors used $10^5$ downsampled images to train their primary models.
  • Smithsonian Butterflies: A much smaller dataset (subset of 500 images) used for ablation studies to see if these generalization properties hold even when training data is scarce.

Key Insights and Ideas

  • Injected Noise is King: The specific sequence of noise added during the process is far more influential than the initial random seed. As long as the injected sequence is shared, models produce consistent results even with different starting noise.
  • The "Speciation" Sweet Spot: The impact of injected noise is not uniform. It is most critical during Regime II (roughly the middle 20-80% of the process). This is the "decision-making" phase where the broad structure is already present, but the specific identity of the sample is being locked in.
  • Early Convergence, Late Speciation: Interestingly, even when models are given different noise sequences, they initially move toward a similar "mean face" in the early steps (Regime I). The divergence into unique identities only happens later.
Figure 2: The injected noise sequence has a much larger impact on final image consistency than the starting noise or the specific model used.

Key Results

  • Quantitative Consistency: When models shared the same injected noise sequence, the average cosine similarity of the final images was 0.91, regardless of the starting noise. When the injected sequence differed, similarity plummeted to 0.02.
  • Temporal Dynamics: In shared-noise scenarios, similarity between intermediate states $x_t$ rises to nearly 1.0 within the first 10% of steps.
  • Regime Sensitivity: The authors found that altering the noise in the middle of the process (40-60% interval) caused the largest drop in similarity, confirming that Regime II is the most sensitive phase.
Figure 1: Sensitivity to noise changes is significantly higher during the middle "Regime II" than in the early or late stages of sampling.

Limitations and Future Work

  • Dataset Size Matters: In experiments with the small butterfly dataset, the clear distinction between regimes was less apparent. The models struggled to denoise effectively in early stages, suggesting that "strong generalization" might require a minimum threshold of training data or model capacity.
  • Fixed Step Count: The study used a fixed $T=250$ steps for easier comparison. Real-world applications often use dynamic stopping criteria based on Signal-to-Noise Ratio (SNR), which might shift these regime boundaries.
  • Scope: The work focused primarily on image generation (faces and butterflies). Future research could investigate if these noise-driven dynamics hold for other modalities like text or audio.

Summary updated

Abstract

Diffusion models trained on distinct datasets can produce highly similar samples when prompted with the same noise input, suggesting that they learn a generalized data distribution. We investigate the role of noise injected during the sampling process, and show that it plays an important role in making or breaking this strong generalization effect. Based on this finding we investigate the robustness of sample similarity to changes in the injected noise, and find that its impact is largest in the sampling regime where broad structure already exists but the precise identity of the sample is not yet determined.
BibTeX
@misc{emmerling2026role,
  author = {Vincent Emmerling and Felix Günther and David Kappel},
  title = {{The role of injected noise in diffusion model generalization}},
  year = {2026},
  eprint = {2611.03764},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03764},
}
Figure