Score Matching Model for Unbounded Data Score

Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, Il-Chul Moon

Abstract

Recent advances in diffusion models have achieved state-of-the-art performance in image generation. However, empirical evidence suggests an inverse correlation between density estimation and sample generation performance. This paper investigates this phenomenon, finding that density estimation is primarily driven by small diffusion times, while sample generation depends on large diffusion times. Training a score network effectively across the entire diffusion time spectrum is challenging due to the significant imbalance in loss scales. To address this, we introduce Soft Truncation, a universally applicable training technique that replaces the fixed truncation hyperparameter with a random variable. Soft Truncation achieves state-of-the-art results on CIFAR-10, CelebA, CelebA-HQ, and STL-10 benchmarks.

The Problem: The Inverse Correlation Gap

In score-based diffusion models, there is a known trade-off between two primary metrics: Density Estimation (Negative Log-Likelihood, NLL) and Sample Generation (Fréchet Inception Distance, FID). The authors identify that this gap arises because these objectives prioritize different stages of the diffusion process.

Density estimation is heavily influenced by small diffusion times (low noise), where the loss scale is massive and tends to dominate training. Conversely, high-quality sample generation relies on accuracy at large diffusion times (high noise) to establish global structure. Standard practice uses a fixed truncation hyperparameter ($\epsilon$) to ignore unstable gradients near zero time. However, this creates a zero-sum game: a small $\epsilon$ favors NLL but degrades FID, while a larger $\epsilon$ improves FID at the expense of NLL.

Methods: Soft Truncation and Unbounded Parametrization

The core innovation, Soft Truncation, treats the truncation point as a random variable $\tau$ rather than a static value.

  • The Algorithm: During each mini-batch update, a new $\tau$ is sampled from a probability distribution. The model then calculates the loss only for the range $[\tau, T]$. This forces the score network to learn across the entire range of diffusion times, preventing it from being overwhelmed by the high-variance loss at the start of the process.
  • Theoretical Foundation: The authors frame this as Maximum Perturbed Likelihood Estimation (MPLE), proving it is mathematically equivalent to training with a general weighting function.
  • Unbounded Parametrization: To stabilize training near $t=0$, the authors introduce "Unbounded" architectures (UNCSN++ and UDDPM++). These use specific input transformations to handle the diverging nature of the data score as noise approaches zero.

Key Insights: Addressing Loss Imbalance

The primary motivation for Soft Truncation is the extreme imbalance in the training loss. The integrand of the variational bound is orders of magnitude larger at $t \approx 0$ than at $t=1$.

Analysis of loss imbalance and its impact on performance. (a) shows the diverging loss term as time approaches zero. (b) and (c) demonstrate how truncation affects the training bound and test performance across different time scales.

Soft Truncation acts as a natural curriculum or regularizer. By varying the distribution of $\tau$ (controlled by a parameter $k$), researchers can interpolate between "likelihood-favorable" and "FID-favorable" regimes. This flexibility is crucial for balancing "microscopic" quality (fine textures) and "macroscopic" quality (global structure). Too much truncation leads to unrealistic textures, while too little prevents the model from learning the big picture.

Microscopic sample quality at very low noise levels ($\sigma=0.001$), representing the regime critical for density estimation.

Experimental Results

Soft Truncation consistently outperforms vanilla training across various architectures (NCSN++, DDPM++) and SDE types (VESDE, VPSDE).

  • CelebA-HQ 256: Achieved a state-of-the-art FID of 1.90, a significant improvement over the previous best of 2.92.
  • CIFAR-10: Reached an FID of 2.33 with UNCSN++, outperforming specialized likelihood-based models while maintaining competitive NLL.
  • STL-10: Reported a major FID improvement to 7.71.
  • Efficiency: The method adds negligible computational overhead and requires no extra parameters, making it a "drop-in" enhancement for existing pipelines.

Conclusion

Soft Truncation resolves the long-standing trade-off between density estimation and sample generation in diffusion models. By treating the truncation time as a random variable, the model learns a more robust score function across all noise levels, leading to superior image quality and competitive likelihoods across multiple benchmarks.

Summary updated

Abstract

Recent advances in diffusion models bring state-of-the-art performance on image generation tasks. However, empirical results from previous research in diffusion models imply an inverse correlation between density estimation and sample generation performances. This paper investigates with sufficient empirical evidence that such inverse correlation happens because density estimation is significantly contributed by small diffusion time, whereas sample generation mainly depends on large diffusion time. However, training a score network well across the entire diffusion time is demanding because the loss scale is significantly imbalanced at each diffusion time. For successful training, therefore, we introduce Soft Truncation, a universally applicable training technique for diffusion models, that softens the fixed and static truncation hyperparameter into a random variable. In experiments, Soft Truncation achieves state-of-the-art performance on CIFAR-10, CelebA, CelebA-HQ 256x256, and STL-10 datasets.
BibTeX
@inproceedings{kim2022score,
  author = {Dongjun Kim and Seungjae Shin and Kyungwoo Song and Wanmo Kang and Il-Chul Moon},
  title = {{Score Matching Model for Unbounded Data Score}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=Fa2RChv8HA},
}
Figure