Abstract
Recent advances in diffusion models bring state-of-the-art performance on image generation tasks. However, empirical results from previous research in diffusion models imply an inverse correlation between density estimation and sample generation performances. This paper investigates with sufficient empirical evidence that such inverse correlation happens because density estimation is significantly contributed by small diffusion time, whereas sample generation mainly depends on large diffusion time. However, training a score network well across the entire diffusion time is demanding because the loss scale is significantly imbalanced at each diffusion time. For successful training, therefore, we introduce Soft Truncation, a universally applicable training technique for diffusion models, that softens the fixed and static truncation hyperparameter into a random variable. In experiments, Soft Truncation achieves state-of-the-art performance on CIFAR-10, CelebA, CelebA-HQ 256x256, and STL-10 datasets.
Introduction
Diffusion models have achieved remarkable success in generative AI, yet they often face a trade-off between sample quality (measured by Fréchet Inception Distance, FID) and density estimation accuracy (measured by Negative Log-Likelihood, NLL). A model optimized for high-fidelity image generation frequently performs poorly at estimating data density, and vice versa. This "split personality" stems from the differing requirements of these two tasks across the diffusion time spectrum.
The Challenge: Loss Imbalance and the FID-NLL Trade-off
The fundamental difficulty in training diffusion models lies in the imbalance of the training process across the time range $[0, T]$. Density estimation (NLL) relies heavily on "microscopic" details learned at very small diffusion times ($t \approx 0$), whereas sample generation (FID) depends more on the "macroscopic" structure learned at larger diffusion times.
As shown in Figure 1, the training loss function diverges as $t$ approaches zero. To maintain stability, researchers typically employ a fixed truncation hyperparameter, $\epsilon$, training the model only on the interval $[\epsilon, T]$. This creates a dilemma: an $\epsilon$ that is too small leads to an imbalanced loss that ruins global structure (poor FID), while an $\epsilon$ that is too large prevents the model from learning fine details (poor NLL).
Proposed Method: Soft Truncation
To resolve this trade-off, we propose Soft Truncation. Instead of a fixed $\epsilon$, the truncation point is treated as a random variable, $\tau$, sampled from a distribution $\mathbb{P}(\tau)$ during each training update. In every step, a new $\tau$ is sampled, and the model calculates the loss for the range $[\tau, T]$.
This approach ensures the score network eventually observes the entire range of diffusion times without being overwhelmed by the massive loss scale near $t=0$ in any single update. Theoretically, Soft Truncation is equivalent to training with a specific "general weighting" function and can be framed as Maximum Perturbed Likelihood Estimation (MPLE), where the randomness helps the model escape local optima.
Experimental Results
Soft Truncation was evaluated across multiple benchmarks, including CIFAR-10, CelebA-HQ, and STL-10. The technique consistently outperformed vanilla training methods, effectively bridging the gap between NLL and FID.
- CIFAR-10: Achieved a state-of-the-art FID of 2.33 while maintaining competitive NLL.
- CelebA-HQ (256x256): Reached an FID of 1.90, a significant improvement over the previous best of 2.92.
- High Precision: The method enables the model to capture extremely fine details at low noise scales, as illustrated in the high-precision reconstruction at $\sigma=0.001$.
Conclusion
Soft Truncation provides a simple, universally applicable "plug-and-play" solution to the loss imbalance problem in diffusion models. By replacing static truncation with a stochastic process, it allows a single model to excel at both high-fidelity image synthesis and precise density estimation. Future work will explore the optimal selection of the distribution for the random variable $\tau$ across different architectures and datasets.
Summary updated