This paper, presented at ICML 2022, introduces a simple yet powerful training modification for diffusion models that resolves a long-standing tension in generative modeling: the trade-off between high-quality image generation and accurate density estimation.
TL;DR
Soft Truncation is a training technique that replaces the fixed "noise floor" (truncation hyperparameter) in diffusion models with a random variable. This allows the model to learn both global structure and fine-grained details effectively, achieving state-of-the-art image quality (FID) without sacrificing likelihood (NLL).
What Problem Does This Paper Address?
Diffusion models work by gradually adding noise to data and then learning to reverse that process. However, researchers have noticed an annoying inverse correlation: models that are great at "density estimation" (assigning high probability to real data, measured by Negative Log-Likelihood or NLL) often produce mediocre images. Conversely, models that produce stunning images (measured by Fréchet Inception Distance or FID) often have poor NLL.
The authors identify the root cause of this trade-off as a loss imbalance across different diffusion times.
- Small diffusion times ($t \approx 0$): The noise is very low, and the model focuses on microscopic details. Because the score function (the gradient of the log-density) diverges as noise goes to zero, the loss scale here is massive. This dominates the training, which is good for NLL but ignores the "big picture."
- Large diffusion times ($t \approx T$): The noise is high, and the model learns the global structure of the image (e.g., "this is a face"). The loss scale here is much smaller, so it often gets drowned out during training.
Standard practice is to use a fixed "truncation" hyperparameter $\epsilon$, ignoring everything below a certain noise level to prevent gradients from exploding. But picking a single $\epsilon$ is a compromise: too small, and you ruin the FID; too large, and you ruin the NLL.
Methods: Soft Truncation
The authors propose Soft Truncation. Instead of picking one fixed $\epsilon$, they treat the truncation point as a random variable $\tau$ sampled from a distribution $P(\tau)$ at every training step.
- The Algorithm: In each mini-batch, the trainer samples a new $\tau$. The model is then optimized to estimate the score function only on the range $[\tau, T]$.
- The Distribution: They parametrize the distribution of $\tau$ using a power law: $P_k(\tau) \propto 1/\tau^k$. By adjusting $k$, they can control how much weight is given to different noise levels.
- Theoretical Grounding: They prove that Soft Truncation is mathematically equivalent to training a diffusion model with a "general weighting" function. They frame this as Maximum Perturbed Likelihood Estimation (MPLE)—a generalization of standard Maximum Likelihood that accounts for varying levels of noise perturbation.
Key Insights and Ideas
- Truncation is a Knob, not just a Hack: Previously, truncation was seen as a numerical necessity to prevent gradients from exploding. This paper shows it is actually the primary hyperparameter controlling the NLL-FID trade-off.
- "Softening" the Constraint: By varying $\tau$, the score network gets "fair" exposure to both large-scale structure (when $\tau$ is large) and fine details (when $\tau$ is small). This prevents the model from over-fitting to the high-magnitude loss at $t \approx 0$.
- The "Aha" Moment on Weighting: While previous works manually designed weighting functions (like "variance weighting" for FID or "likelihood weighting" for NLL), Soft Truncation implicitly creates an optimal weighting function just by changing the integration bounds.
Datasets and Key Results
The authors evaluated Soft Truncation across several standard benchmarks, including CIFAR-10, ImageNet 32×32, CelebA (64×64), CelebA-HQ (256×256), STL-10 (48×48), and LSUN Bedroom & Church.
Soft Truncation achieved state-of-the-art (SOTA) results at the time of publication:
- CIFAR-10: Achieved an FID of 1.90, outperforming the previous best (DDGM at 2.92) while maintaining competitive NLL.
- CelebA-HQ 256: Achieved an FID of 7.16.
- STL-10: Significantly improved FID to 7.71.
- Universal Applicability: The authors showed the technique works across different architectures (NCSN++, DDPM++) and different types of Stochastic Differential Equations (VESDE, VPSDE).
Qualitatively, the models trained with Soft Truncation showed much better "global fidelity"—avoiding artifacts like unrealistic hair textures or warped features that occur when a model ignores large-time diffusion steps.
Limitations and Future Work
- Distribution Selection: While the $1/\tau^k$ distribution works well, the authors acknowledge that they haven't found a "theoretically optimal" distribution for $\tau$. Finding the best $P(\tau)$ for a specific dataset still requires some empirical searching (e.g., they found $k=1.0$ best for some settings and $k=2.0$ for others).
- Computational Overhead: While the training modification is simple, the paper notes that importance sampling is necessary to keep training efficient, and the interaction between Soft Truncation and different samplers (like ODE vs. PC samplers) could be explored further.
- Discrete Models: The focus was primarily on continuous-time SDE formulations; extending the rigorous MPLE framework to purely discrete-time models is a suggested path for future work.
Summary updated