Introduction
Video diffusion models have revolutionized the generation of high-fidelity, photo-realistic video content. However, their deployment remains a significant challenge due to extreme computational requirements. These models typically demand massive VRAM and suffer from long inference times, often making them inaccessible for consumer-grade hardware.
Traditionally, optimizing these models for different hardware constraints—such as moving from a high-end server to a mobile device—requires a costly trial-and-error process of designing and retraining separate architectures. Furthermore, high-resolution video generation often relies on a multi-stage pipeline of base and super-resolution models, which compounds memory overhead. To address these inefficiencies, this paper introduces SNED (Superposition Network Architecture Search), a unified framework that utilizes a weight-sharing supernet to support multiple resolutions and hardware constraints within a single trained model.
Methodology
SNED leverages a One-Shot Neural Architecture Search (NAS) approach specifically tailored for the complexities of video diffusion. Instead of training multiple independent models, the framework trains a single "supernet" that contains a vast space of smaller, nested "subnets."
Weight-Sharing Supernet
The core of SNED is the weight-sharing mechanism. Smaller subnets are embedded within the larger supernet, sharing the same parameters. This allows users to "slice" a specific portion of the supernet to create a model that fits a particular parameter budget or resolution requirement without needing to retrain from scratch.
Dynamic Search Space
The search space is optimized across two primary dimensions:
- Dynamic Channels: The width of the layers can be adjusted (e.g., utilizing between 40% and 100% of the original channels).
- Fine-grained Dynamic Blocks: The architecture can dynamically decide whether to retain or drop specific components within a diffusion block, such as temporal self-attention or cross-attention layers.
Super-position Training and Sampling Warm-up
A novel aspect of this framework is Super-position Training. During the training phase, the model is simultaneously exposed to data at various resolutions (e.g., 64x64, 128x128, and 256x256). This forces the shared weights to learn multi-scale features that are robust across different output sizes. To maintain training stability—a common issue in NAS for diffusion—the authors implement a Sampling Warm-up strategy. This involves training the full supernet initially and gradually introducing the random sampling of smaller subnets over 30,000 iterations.
Experimental Evaluation
The researchers evaluated SNED across two distinct architectures:
- Pixel-Space Models: Trained on an extensive internal dataset comprising 19 million video-text pairs.
- Latent-Space Models (LVDM): Evaluated on the Sky Timelapse dataset, a standard benchmark for unconditional video generation.
Key Results
- Efficiency and Scalability: In pixel-space experiments, SNED successfully managed subnets ranging from 640M to 1.6B parameters, maintaining high video quality across resolutions from 64x64 to 256x256.
- Inference Speedup: On the Sky Timelapse benchmark, the smallest SNED subnet (274M parameters) achieved a 1.53x speedup in inference latency compared to the baseline LVDM model while maintaining comparable visual quality.
- Quality Improvements: Interestingly, the search process often identified subnets that outperformed the full supernet in terms of Fréchet Video Distance (FVD) and Kernel Video Distance (KVD). This suggests that the NAS process effectively prunes redundant or noisy components, finding an optimal balance between model complexity and data representation.
Discussion and Limitations
SNED demonstrates that a single set of weights can be adapted to various output sizes and hardware constraints, drastically reducing the total memory footprint of model deployment pipelines. However, the initial training of the supernet remains computationally intensive; the pixel-space experiments required 64 A100 GPUs for 140,000 iterations. Additionally, while the results on the internal 19M dataset are impressive, the lack of public access to this data poses a challenge for exact replication by the broader research community.
Conclusion
SNED represents a significant advancement in making video diffusion models more practical for real-world deployment. By replacing fragmented, per-device training with a unified "super-position" framework, SNED provides a scalable path toward efficient, high-quality video generation across a wide range of hardware environments.
Summary updated