Introduction
Diffusion Models (DMs) have become the gold standard for generative AI, yet they are notoriously difficult and slow to train. Most DMs rely on the UNet architecture, which uses Long Skip Connections (LSCs) to pass information from early encoder layers directly to later decoder layers. While LSCs help aggregate long-distance information and prevent vanishing gradients, they also introduce a hidden cost: instability.
Researchers have identified that hidden features and gradients in deep UNets often oscillate wildly during training. While some previous work found that scaling these connections by a constant factor (like $1/\sqrt{2}$) helped, there was no theoretical explanation for why this worked or how to do it optimally. ScaleLong fills this gap by providing a rigorous mathematical analysis of UNet stability and proposing a more principled way to scale these connections.
Methods: The ScaleLong Framework
The authors introduce ScaleLong, a framework designed to stabilize training by intelligently scaling the LSC coefficients ($\kappa_i$). They propose two specific strategies:
- Constant Scaling (CS): Instead of scaling every connection by the same factor, CS uses an exponentially decaying series: $\kappa_i = \kappa^{i-1}$ (where $\kappa \in (0, 1]$). This ensures that connections spanning more blocks are scaled down more aggressively, which the authors prove is more effective at reducing the "robustness error bound" than universal scaling.
- Learnable Scaling (LS): This method uses a tiny, shared "calibration network" (similar to a Squeeze-and-Excitation block) to predict the optimal scaling coefficient for each LSC based on the input features. This allows the model to adaptively adjust its internal information flow during training.
Crucially, these methods are "plug-and-play" and require minimal code changes to existing UNet or UViT (Transformer-based UNet) architectures.
Theoretical Insights and Stability
The paper proves that in a standard UNet, the oscillation range of hidden features is proportional to the number of LSCs ($N$) and the feature dimension ($m$). In deep networks, this range becomes massive, leading to the "feature oscillation" that plagues training.
Furthermore, theoretical analysis shows that standard UNets are provably sensitive to small perturbations in input. Since diffusion models are essentially iterative denoisers, this sensitivity leads to oscillatory losses and gradients, slowing down the "learning" process. One of the most interesting findings is that scaling coefficients should decay exponentially. This is because the stability bound is dominated by terms that grow with the number of blocks. By using $\kappa^{i-1}$, the model effectively "muffles" the most disruptive connections while keeping the network's representational power intact. Empirically, when the researchers trained the Learnable Scaling (LS) version, they found that the model naturally learned a decaying pattern very similar to the manual Constant Scaling (CS) formula.
Experimental Results
The authors evaluated ScaleLong across four diverse benchmarks: CIFAR10, CelebA, ImageNet (64x64), and MS-COCO.
- 1.5x Training Acceleration: Across different datasets and architectures (UNet and UViT), ScaleLong reached high-quality results significantly faster than standard training. For example, on a 32-batch UViT setup, LS reached superior performance in 6.7 hours compared to 15.6 hours for the baseline.
- Improved Generation Quality: The methods consistently achieved lower (better) FID scores. On CIFAR10, UViT with CS achieved an FID of 2.98, compared to 3.11 for the baseline.
- Robustness to Noise: The authors demonstrated that ScaleLong models are much more resilient to "extra noise" (perturbations) added during training, which is a common real-world challenge in large-scale data pipelines.
Conclusion
ScaleLong provides a principled solution to the training instabilities inherent in deep UNet-based diffusion models. By scaling long skip connections, the framework achieves a significant training speedup (roughly 1.5x) and improved generation quality. While the analysis focuses on standard UNet and UViT structures, these stability principles offer a promising direction for optimizing even more complex generative architectures in the future.
Summary updated