Introduction
Generating high-quality video using Diffusion Probabilistic Models (DPMs) presents significant challenges. Standard DPMs typically treat video frames as independent noise samples, which ignores the inherent temporal redundancy of video data. This approach places a heavy computational burden on denoising networks and often results in temporal inconsistencies, such as flickering or identity drift. VideoFusion addresses these issues by decomposing the diffusion process into global content and local motion components.
The Decomposed Diffusion Process
The fundamental innovation of VideoFusion is the decomposition of the diffusion noise. Instead of adding independent noise to each frame, the authors split the noise into a shared base noise ($b_t$) and a frame-specific residual noise ($r_t^i$).
As shown in the diagram, the base noise represents the common global content (the "base frame"), while the residual noise captures the motion and changes between frames. This decomposition allows the model to leverage strong image priors while efficiently modeling temporal dynamics.
Architecture and Training
VideoFusion utilizes a dual-generator architecture:
- Base Generator: A large, pre-trained image diffusion model that processes a single base frame per step. This provides high-quality textures and structural priors.
- Residual Generator: A smaller, 2D U-Net that predicts frame-specific residuals, conditioned on the frame index to model motion.
The model is trained using a joint strategy with a "stop-gradient" mechanism. By preventing gradients from the initially noisy residual generator from affecting the pre-trained base generator, the system preserves the high-quality image generation capabilities of the base model during the early stages of video-specific training.
Key Insights: Content vs. Motion Control
The decomposition of noise provides unique control over the generated content. By fixing the base noise and varying the residual noise, the model can generate different motions for the same subject. Conversely, fixing the residual noise while changing the base noise allows for the same motion to be applied to different subjects.
This approach ensures long-term coherence. In tests, fixing the base noise across sequences of up to 512 frames prevented the identity shifting common in other generative methods. Furthermore, because the heavy base generator only requires one forward pass per step for an entire clip, the model is significantly more efficient than standard 3D diffusion models.
Experimental Results
VideoFusion was evaluated on benchmarks including UCF101, Sky Time-lapse, and WebVid-10M. On UCF101, it achieved a state-of-the-art Inception Score (IS) of 72.22 and a Frechet Video Distance (FVD) of 220. Beyond quality improvements, VideoFusion demonstrated substantial efficiency gains over standard 3D diffusion (VDM), reducing memory consumption by 21.8% and improving inference latency by 57.5%.
Conclusion and Future Work
VideoFusion represents a significant step forward in video generation by intelligently exploiting the redundancy of video frames. By decomposing diffusion into base and residual components, it achieves superior quality and efficiency while maintaining temporal coherence. Future work may involve adaptive decomposition ratios to better handle varying levels of motion across different video types.
Summary updated