Critical windows: non-asymptotic theory for feature emergence in diffusion models

Marvin Li, Sitan Chen

Diffusion models generate images through a continuous denoising process, transforming pure noise into structured data. Yet, empirically, they appear to make discrete "decisions" about specific image features—such as object class or background color—during narrow, specific time intervals known as critical windows. While these windows are highly useful for interpretability, allowing researchers to localize exactly when a feature emerges, a rigorous mathematical explanation has historically been lacking. Previous theories were largely confined to simple "toy" models. This paper bridges that gap by introducing a non-asymptotic theory that formally explains critical windows for a much richer family of data distributions.

The Geometry of Critical Windows

To formalize this phenomenon, the authors model the underlying data distribution as a mixture of strongly log-concave densities. This flexible mathematical approach represents complex data as distinct "clusters," where different clusters correspond to different semantic features (e.g., one cluster for "red cars," another for "blue cars").

To analyze feature emergence, the study employs a "noise-then-denoise" experiment: a sample from a specific sub-group is subjected to forward diffusion (adding noise) for time $t$, followed by the reverse denoising process back to the start. This experiment reveals two critical timestamps that define the window:

  • $T_{lower}$: The earliest time at which the added noise causes the model to "forget" the specific starting sample while still retaining the general category (e.g., forgetting the specific cat but remembering it is a cat).
  • $T_{upper}$: The latest time before the noise overwhelms the category entirely, causing the model to lose track of the original group and generate samples from the rest of the distribution.

The interval $[T_{lower}, T_{upper}]$ constitutes the critical window. The authors provide provable bounds for these times based on the geometric separation between different groups in the data. The length of a critical window is intrinsically linked to the "salience" of a feature—specifically, the ratio between the distance separating different classes and the internal spread of data within a single class.

Hierarchical Sampling in Action

A key insight from this framework is that diffusion models function as hierarchical samplers. Rather than deciding all image attributes simultaneously, the model makes a sequence of choices over a discrete sequence of times, progressively whittling away distant possibilities. Larger, more salient features (like the general layout of an object) have earlier critical windows than fine-grained details (like the exact shade of paint).

A diffusion trajectory illustrating hierarchical sampling through data clusters.

Practical Applications: Fairness and Privacy

The theoretical bounds were validated using synthetic Gaussian mixtures and real-world models like Stable Diffusion v2.1 and CIFAR-10, revealing significant practical applications.

Fairness Auditing: By identifying the critical window for a biased or sensitive feature, developers can target interventions. For instance, in experiments with Stable Diffusion, the authors identified a sharp decision point for gender in photo portraits between timesteps 80 and 84 (where agreement with the original image's gender dropped from 80% to 50%). Intervening during this specific interval could improve diversity without disrupting the rest of the generation process.

Privacy as "Persistence": The study also links critical windows to data memorization. Training samples tend to "survive" a noise-denoise cycle longer than non-training samples. Leveraging this persistence, the authors introduce NoiseDenoise, a novel Membership Inference Attack (MIA). While it does not universally outperform specialized state-of-the-art attacks in overall accuracy, it successfully identifies unique subsets of memorized training points that other attacks miss, serving as a valuable complementary tool for privacy auditing.

Limitations and Future Directions

While this framework provides a robust foundation, several challenges remain. The current theoretical bound for $T_{upper}$ exhibits a logarithmic dependence on dimensionality ($d$), which future work aims to eliminate to make the theory more robust for high-resolution images. Additionally, the framework currently focuses on discrete features (classes or bins). Extending this to continuous features (like exact angles or lighting) and testing these bounds on more complex, non-log-concave data manifolds are logical next steps for the field.

Summary updated

Abstract

We develop theory to understand an intriguing property of diffusion models for image generation that we term *critical windows*. Empirically, it has been observed that there are narrow time intervals in sampling during which particular features of the final image emerge, e.g. the image class or background color (Ho et al., 2020b; Meng et al., 2022; Choi et al., 2022; Raya & Ambrogioni, 2023; Georgiev et al., 2023; Sclocchi et al., 2024; Biroli et al., 2024). While this is advantageous for interpretability as it implies one can localize properties of the generation to a small segment of the trajectory, it seems at odds with the continuous nature of the diffusion. We propose a formal framework for studying these windows and show that for data coming from a mixture of strongly log-concave densities, these windows can be provably bounded in terms of certain measures of inter- and intra-group separation. We also instantiate these bounds for concrete examples like well-conditioned Gaussian mixtures. Finally, we use our bounds to give a rigorous interpretation of diffusion models as hierarchical samplers that progressively “decide” output features over a discrete sequence of times. We validate our bounds with experiments on synthetic data and show that critical windows may serve as a useful tool for diagnosing fairness and privacy violations in real-world diffusion models.
BibTeX
@inproceedings{li2024critical,
  author = {Marvin Li and Sitan Chen},
  title = {{Critical windows: non-asymptotic theory for feature emergence in diffusion models}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=a8ZpjLJuKk},
}
Figure