Abstract
Recent advances in diffusion models have achieved state-of-the-art performance in image generation. However, empirical evidence suggests an inverse correlation between density estimation and sample generation performance. This paper investigates this phenomenon, finding that density estimation is primarily driven by small diffusion times, while sample generation depends on large diffusion times. Training a score network effectively across the entire diffusion time spectrum is challenging due to the significant imbalance in loss scales. To address this, we introduce Soft Truncation, a universally applicable training technique that replaces the fixed truncation hyperparameter with a random variable. Soft Truncation achieves state-of-the-art results on CIFAR-10, CelebA, CelebA-HQ, and STL-10 benchmarks.
The Problem: The Inverse Correlation Gap
In score-based diffusion models, there is a known trade-off between two primary metrics: Density Estimation (Negative Log-Likelihood, NLL) and Sample Generation (Fréchet Inception Distance, FID). The authors identify that this gap arises because these objectives prioritize different stages of the diffusion process.
Density estimation is heavily influenced by small diffusion times (low noise), where the loss scale is massive and tends to dominate training. Conversely, high-quality sample generation relies on accuracy at large diffusion times (high noise) to establish global structure. Standard practice uses a fixed truncation hyperparameter ($\epsilon$) to ignore unstable gradients near zero time. However, this creates a zero-sum game: a small $\epsilon$ favors NLL but degrades FID, while a larger $\epsilon$ improves FID at the expense of NLL.
Methods: Soft Truncation and Unbounded Parametrization
The core innovation, Soft Truncation, treats the truncation point as a random variable $\tau$ rather than a static value.
- The Algorithm: During each mini-batch update, a new $\tau$ is sampled from a probability distribution. The model then calculates the loss only for the range $[\tau, T]$. This forces the score network to learn across the entire range of diffusion times, preventing it from being overwhelmed by the high-variance loss at the start of the process.
- Theoretical Foundation: The authors frame this as Maximum Perturbed Likelihood Estimation (MPLE), proving it is mathematically equivalent to training with a general weighting function.
- Unbounded Parametrization: To stabilize training near $t=0$, the authors introduce "Unbounded" architectures (UNCSN++ and UDDPM++). These use specific input transformations to handle the diverging nature of the data score as noise approaches zero.
Key Insights: Addressing Loss Imbalance
The primary motivation for Soft Truncation is the extreme imbalance in the training loss. The integrand of the variational bound is orders of magnitude larger at $t \approx 0$ than at $t=1$.
Soft Truncation acts as a natural curriculum or regularizer. By varying the distribution of $\tau$ (controlled by a parameter $k$), researchers can interpolate between "likelihood-favorable" and "FID-favorable" regimes. This flexibility is crucial for balancing "microscopic" quality (fine textures) and "macroscopic" quality (global structure). Too much truncation leads to unrealistic textures, while too little prevents the model from learning the big picture.
Experimental Results
Soft Truncation consistently outperforms vanilla training across various architectures (NCSN++, DDPM++) and SDE types (VESDE, VPSDE).
- CelebA-HQ 256: Achieved a state-of-the-art FID of 1.90, a significant improvement over the previous best of 2.92.
- CIFAR-10: Reached an FID of 2.33 with UNCSN++, outperforming specialized likelihood-based models while maintaining competitive NLL.
- STL-10: Reported a major FID improvement to 7.71.
- Efficiency: The method adds negligible computational overhead and requires no extra parameters, making it a "drop-in" enhancement for existing pipelines.
Conclusion
Soft Truncation resolves the long-standing trade-off between density estimation and sample generation in diffusion models. By treating the truncation time as a random variable, the model learns a more robust score function across all noise levels, leading to superior image quality and competitive likelihoods across multiple benchmarks.
Summary updated