In the world of diffusion models, researchers often treat the choice of what the model predicts—whether it's the clean data ($x$), the added noise ($\epsilon$), or the velocity ($v$)—as a matter of convenience. Mathematically, these targets are just affine transformations of each other. However, in practice, these choices lead to wildly different results in terms of image quality and training stability.
A new paper from Dongwoo Kim at POSTECH, "Analyzing Learning-Dynamics Across Loss and Prediction Choices for Diffusion Models," provides a rigorous framework to understand why. By disentangling the prediction space (what the network outputs) from the loss space (where the error is measured), the author reveals how architectural bottlenecks and error amplification dictate the success of generative models.
TL;DR
The paper proves that while changing the loss space is simply a form of timestep reweighting, the choice of prediction space is fundamental: $x$-prediction can exploit the low-dimensional structure of data to bypass architectural bottlenecks that impose a hard "error floor" on noise ($\epsilon$) and velocity ($v$) prediction.
What Problem Does This Paper Address?
Diffusion models work by learning to reverse a corruption process. We have nine possible combinations of training: we can predict $\{x, \epsilon, v\}$ and measure the squared error in $\{x, \epsilon, v\}$ space.
While these are algebraically equivalent in an "oracle" regime (infinite model capacity), they are not equivalent in finite-capacity neural networks. Recent high-performing architectures like the pixel-space Transformer (JiT) show a massive preference for $x$-prediction. This paper seeks to explain why this asymmetry exists and how architectural constraints like bottlenecks interact with these choices.
Methods
The author analyzes the $3 \times 3$ grid of prediction vs. loss choices through three lenses:
- Timestep Reweighting: The paper proves that for any fixed prediction target, changing the loss space is mathematically equivalent to applying a specific weight to different timesteps ($t$).
- Bottleneck Analysis: The author models architectural compression (like patch embeddings in Transformers) as a rank-$m$ linear bottleneck. They derive a theoretical "error floor" for models trying to predict high-dimensional noise through such a bottleneck.
- Error Conversion: Since we ultimately care about the clean data $x$, the paper analyzes how errors in $\epsilon$ or $v$ space are amplified when converted back to $x$ space.
Datasets
The theoretical predictions were validated using:
- Synthetic Data: A low-dimensional manifold in high-dimensional space to test bottleneck theories.
- CIFAR-10: A standard benchmark for small-scale image generation.
- Imagenette (64x64): A subset of ImageNet used for rapid iteration.
- ImageNet-256: To confirm that the findings scale to high-resolution, state-of-the-art generative modeling.
Key Insights and Ideas
- Loss Space is just a Volume Knob: Changing the loss space doesn't change what the model learns to predict, only when it tries hardest. For example, using an $\epsilon$-loss while predicting $x$ effectively forces the model to focus more on high-noise timesteps.
- The Bottleneck Asymmetry: This is the "aha" moment. Natural images ($x$) live on low-dimensional manifolds. A network with a bottleneck can still represent $x$ well because it only needs to preserve those few dimensions. However, Gaussian noise ($\epsilon$) is truly high-dimensional—it fills every corner of the space. A bottleneck must discard information about the noise, leading to an irreducible error floor of $D-m$ (where $D$ is the total dimension and $m$ is the bottleneck rank).
- The Conversion Trap: Predicting noise is dangerous at low noise levels. As $t \to 0$, the factor used to convert a noise estimate to a data estimate $(\frac{1-t}{t})$ blows up. This means even a tiny error in noise prediction becomes a massive error in the final image.
The implicit weights for different combinations are summarized in the paper's first table:
Key Results
The paper's experiments provide striking confirmation of the theory:
- Bottleneck Validation: In synthetic tests, the error for $\epsilon$-prediction matched the theoretical $D-m$ floor almost perfectly, while $x$-prediction maintained low error as long as the bottleneck was wider than the data's intrinsic dimension.
- JiT Performance: On Imagenette, a compressed JiT model (ratio 0.67) using $x$-prediction achieved an MSE of ~0.048. In contrast, $\epsilon$-prediction on the same architecture resulted in an MSE of 5.85—over 120 times worse.
- Qualitative Collapse: Figure 3 demonstrates the impact of these choices. Under a bottleneck, $x$-prediction still produces recognizable images, while $\epsilon$-prediction collapses into pure noise because the network literally cannot "see" enough of the noise to subtract it.
- U-Net vs. Transformer: For standard U-Nets (which have skip connections that bypass bottlenecks), the choice of loss space matters more than the prediction space. Switching to $\epsilon$-loss or $v$-loss significantly improved FID (sample quality) by emphasizing the "generation-critical" high-noise regime.
Limitations and Future Work
- Linearity Assumption: The core proof assumes a linear bottleneck. While the authors show empirically that adding non-linearities (like GeLU) doesn't break the trend, a formal non-linear theory is still missing.
- FID vs. MSE: While the paper explains the MSE (denoising accuracy) gap well, it notes that the FID (perceptual quality) gap persists even when MSE is similar, suggesting there are still nuances in the "learning difficulty" of different targets that aren't fully captured by information theory alone.
- Scope: The study focuses on pixel-space models. Extending this analysis to Latent Diffusion Models (LDM) or models with explicit skip-connection conditioning (like EDM) is a logical next step.
Summary updated