SNED: Superposition Network Architecture Search for Efficient Video Diffusion Model

Zhengang Li, Yan Kang, Yuchen Liu, Difan Liu, Tobias Hinz, Feng Liu, Yanzhi Wang

Introduction

Video diffusion models have revolutionized the generation of high-fidelity, photo-realistic video content. However, their deployment remains a significant challenge due to extreme computational requirements. These models typically demand massive VRAM and suffer from long inference times, often making them inaccessible for consumer-grade hardware.

Traditionally, optimizing these models for different hardware constraints—such as moving from a high-end server to a mobile device—requires a costly trial-and-error process of designing and retraining separate architectures. Furthermore, high-resolution video generation often relies on a multi-stage pipeline of base and super-resolution models, which compounds memory overhead. To address these inefficiencies, this paper introduces SNED (Superposition Network Architecture Search), a unified framework that utilizes a weight-sharing supernet to support multiple resolutions and hardware constraints within a single trained model.

Methodology

SNED leverages a One-Shot Neural Architecture Search (NAS) approach specifically tailored for the complexities of video diffusion. Instead of training multiple independent models, the framework trains a single "supernet" that contains a vast space of smaller, nested "subnets."

Weight-Sharing Supernet

The core of SNED is the weight-sharing mechanism. Smaller subnets are embedded within the larger supernet, sharing the same parameters. This allows users to "slice" a specific portion of the supernet to create a model that fits a particular parameter budget or resolution requirement without needing to retrain from scratch.

Dynamic Search Space

The search space is optimized across two primary dimensions:

  1. Dynamic Channels: The width of the layers can be adjusted (e.g., utilizing between 40% and 100% of the original channels).
  2. Fine-grained Dynamic Blocks: The architecture can dynamically decide whether to retain or drop specific components within a diffusion block, such as temporal self-attention or cross-attention layers.

Super-position Training and Sampling Warm-up

A novel aspect of this framework is Super-position Training. During the training phase, the model is simultaneously exposed to data at various resolutions (e.g., 64x64, 128x128, and 256x256). This forces the shared weights to learn multi-scale features that are robust across different output sizes. To maintain training stability—a common issue in NAS for diffusion—the authors implement a Sampling Warm-up strategy. This involves training the full supernet initially and gradually introducing the random sampling of smaller subnets over 30,000 iterations.

Experimental Evaluation

The researchers evaluated SNED across two distinct architectures:

  1. Pixel-Space Models: Trained on an extensive internal dataset comprising 19 million video-text pairs.
  2. Latent-Space Models (LVDM): Evaluated on the Sky Timelapse dataset, a standard benchmark for unconditional video generation.

Key Results

  • Efficiency and Scalability: In pixel-space experiments, SNED successfully managed subnets ranging from 640M to 1.6B parameters, maintaining high video quality across resolutions from 64x64 to 256x256.
  • Inference Speedup: On the Sky Timelapse benchmark, the smallest SNED subnet (274M parameters) achieved a 1.53x speedup in inference latency compared to the baseline LVDM model while maintaining comparable visual quality.
  • Quality Improvements: Interestingly, the search process often identified subnets that outperformed the full supernet in terms of Fréchet Video Distance (FVD) and Kernel Video Distance (KVD). This suggests that the NAS process effectively prunes redundant or noisy components, finding an optimal balance between model complexity and data representation.

Discussion and Limitations

SNED demonstrates that a single set of weights can be adapted to various output sizes and hardware constraints, drastically reducing the total memory footprint of model deployment pipelines. However, the initial training of the supernet remains computationally intensive; the pixel-space experiments required 64 A100 GPUs for 140,000 iterations. Additionally, while the results on the internal 19M dataset are impressive, the lack of public access to this data poses a challenge for exact replication by the broader research community.

Conclusion

SNED represents a significant advancement in making video diffusion models more practical for real-world deployment. By replacing fragmented, per-device training with a unified "super-position" framework, SNED provides a scalable path toward efficient, high-quality video generation across a wide range of hardware environments.

Summary updated

Abstract

While AI-generated content has garnered significant attention, achieving photo-realistic video synthesis remains a formidable challenge. Despite the promising advances in diffusion models for video generation quality, the complex model architecture and substantial computational demands for both training and inference create a significant gap between these models and real-world applications. This paper presents SNED, a superposition network architecture search method for efficient video diffusion model. Our method employs a supernet training paradigm that targets various model cost and resolution options using a weight-sharing method. Moreover, we propose the supernet training sampling warm-up for fast training optimization. To showcase the flexibility of our method, we conduct experiments involving both pixel-space and latent-space video diffusion models. The results demonstrate that our framework consistently produces comparable results across different model options with high efficiency. According to the experiment for the pixel-space video diffusion model, we can achieve consistent video generation results simultaneously across 64×64 to 256×256 resolutions with a large range of model sizes from 640M to 1.6B number of parameters for pixel-space video diffusion models.
BibTeX
@inproceedings{li2024sned,
  author = {Zhengang Li and Yan Kang and Yuchen Liu and Difan Liu and Tobias Hinz and Feng Liu and Yanzhi Wang},
  title = {{SNED: Superposition Network Architecture Search for Efficient Video Diffusion Model}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure