Analyzing Learning-Dynamics Across Loss and Prediction Choices for Diffusion Models

Dongwoo KimarXiv 2026 · arXiv

In the world of diffusion models, researchers often treat the choice of what the model predicts—whether it's the clean data ($x$), the added noise ($\epsilon$), or the velocity ($v$)—as a matter of convenience. Mathematically, these targets are just affine transformations of each other. However, in practice, these choices lead to wildly different results in terms of image quality and training stability.

A new paper from Dongwoo Kim at POSTECH, "Analyzing Learning-Dynamics Across Loss and Prediction Choices for Diffusion Models," provides a rigorous framework to understand why. By disentangling the prediction space (what the network outputs) from the loss space (where the error is measured), the author reveals how architectural bottlenecks and error amplification dictate the success of generative models.

TL;DR

The paper proves that while changing the loss space is simply a form of timestep reweighting, the choice of prediction space is fundamental: $x$-prediction can exploit the low-dimensional structure of data to bypass architectural bottlenecks that impose a hard "error floor" on noise ($\epsilon$) and velocity ($v$) prediction.

What Problem Does This Paper Address?

Diffusion models work by learning to reverse a corruption process. We have nine possible combinations of training: we can predict $\{x, \epsilon, v\}$ and measure the squared error in $\{x, \epsilon, v\}$ space.

While these are algebraically equivalent in an "oracle" regime (infinite model capacity), they are not equivalent in finite-capacity neural networks. Recent high-performing architectures like the pixel-space Transformer (JiT) show a massive preference for $x$-prediction. This paper seeks to explain why this asymmetry exists and how architectural constraints like bottlenecks interact with these choices.

Methods

The author analyzes the $3 \times 3$ grid of prediction vs. loss choices through three lenses:

  1. Timestep Reweighting: The paper proves that for any fixed prediction target, changing the loss space is mathematically equivalent to applying a specific weight to different timesteps ($t$).
  2. Bottleneck Analysis: The author models architectural compression (like patch embeddings in Transformers) as a rank-$m$ linear bottleneck. They derive a theoretical "error floor" for models trying to predict high-dimensional noise through such a bottleneck.
  3. Error Conversion: Since we ultimately care about the clean data $x$, the paper analyzes how errors in $\epsilon$ or $v$ space are amplified when converted back to $x$ space.

Datasets

The theoretical predictions were validated using:

  • Synthetic Data: A low-dimensional manifold in high-dimensional space to test bottleneck theories.
  • CIFAR-10: A standard benchmark for small-scale image generation.
  • Imagenette (64x64): A subset of ImageNet used for rapid iteration.
  • ImageNet-256: To confirm that the findings scale to high-resolution, state-of-the-art generative modeling.

Key Insights and Ideas

  • Loss Space is just a Volume Knob: Changing the loss space doesn't change what the model learns to predict, only when it tries hardest. For example, using an $\epsilon$-loss while predicting $x$ effectively forces the model to focus more on high-noise timesteps.
  • The Bottleneck Asymmetry: This is the "aha" moment. Natural images ($x$) live on low-dimensional manifolds. A network with a bottleneck can still represent $x$ well because it only needs to preserve those few dimensions. However, Gaussian noise ($\epsilon$) is truly high-dimensional—it fills every corner of the space. A bottleneck must discard information about the noise, leading to an irreducible error floor of $D-m$ (where $D$ is the total dimension and $m$ is the bottleneck rank).
  • The Conversion Trap: Predicting noise is dangerous at low noise levels. As $t \to 0$, the factor used to convert a noise estimate to a data estimate $(\frac{1-t}{t})$ blows up. This means even a tiny error in noise prediction becomes a massive error in the final image.

The implicit weights for different combinations are summarized in the paper's first table:

Table 1: The implicit timestep weights ws|u(t) induced by different loss-space choices, derived from the linear-schedule relations between (x, ϵ, v) collected from prior work [Salimans and Ho, 2022, Kingma and Gao, 2023, Karras et al., 2022]. Diagonal entries (u=s) are unity. Off-diagonal entries involving (1−t)−1 or t−1 diverge at endpoints and require clamping in practice.

Key Results

The paper's experiments provide striking confirmation of the theory:

  • Bottleneck Validation: In synthetic tests, the error for $\epsilon$-prediction matched the theoretical $D-m$ floor almost perfectly, while $x$-prediction maintained low error as long as the bottleneck was wider than the data's intrinsic dimension.
  • JiT Performance: On Imagenette, a compressed JiT model (ratio 0.67) using $x$-prediction achieved an MSE of ~0.048. In contrast, $\epsilon$-prediction on the same architecture resulted in an MSE of 5.85—over 120 times worse.
  • Qualitative Collapse: Figure 3 demonstrates the impact of these choices. Under a bottleneck, $x$-prediction still produces recognizable images, while $\epsilon$-prediction collapses into pure noise because the network literally cannot "see" enough of the noise to subtract it.
Figure 3: Qualitative comparison of samples under a bottleneck. x-prediction remains robust while epsilon-prediction collapses.
  • U-Net vs. Transformer: For standard U-Nets (which have skip connections that bypass bottlenecks), the choice of loss space matters more than the prediction space. Switching to $\epsilon$-loss or $v$-loss significantly improved FID (sample quality) by emphasizing the "generation-critical" high-noise regime.

Limitations and Future Work

  • Linearity Assumption: The core proof assumes a linear bottleneck. While the authors show empirically that adding non-linearities (like GeLU) doesn't break the trend, a formal non-linear theory is still missing.
  • FID vs. MSE: While the paper explains the MSE (denoising accuracy) gap well, it notes that the FID (perceptual quality) gap persists even when MSE is similar, suggesting there are still nuances in the "learning difficulty" of different targets that aren't fully captured by information theory alone.
  • Scope: The study focuses on pixel-space models. Extending this analysis to Latent Diffusion Models (LDM) or models with explicit skip-connection conditioning (like EDM) is a logical next step.

Summary updated

Abstract

Denoising generative models predict clean data ($x$), noise ($ε$), or velocity ($v$), while training losses can be measured in any of these spaces. Although the resulting nine combinations are algebraically related, they can yield substantially different denoising accuracy and sample quality. We disentangle the effects of prediction and loss spaces. For a fixed prediction target, we show that changing the loss space is exactly equivalent to timestep reweighting and provide a heuristic analysis of how this reweighting affects generation quality. We then show that a rank-$m$ linear bottleneck compressing a $D$-dimensional noisy input imposes an irreducible squared-error floor of $D-m$ on noise and velocity prediction, whereas clean-data prediction can exploit low-dimensional data structure. Combining this bound with prediction-dependent error conversion yields distinct necessary conditions on bottleneck capacity. Experiments with U-Net and JiT across CIFAR-10, Imagenette, and ImageNet-256 support these theoretical predictions. Under compression, prediction choice strongly affects both denoising accuracy and sample quality, with noise-prediction errors closely matching the theoretical lower bound. Our results provide a quantitative understanding of how architectural bottlenecks interact with prediction and loss choices in denoising generative models.
BibTeX
@misc{kim2026analyzing,
  author = {Dongwoo Kim},
  title = {{Analyzing Learning-Dynamics Across Loss and Prediction Choices for Diffusion Models}},
  year = {2026},
  eprint = {2611.17967},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17967},
}
Figure