Abstract
Generating high-fidelity video remains a significant challenge due to the high computational costs associated with the temporal dimension. We present COMUNI, a novel two-stage diffusion framework that addresses this by decomposing video signals into Common (static appearance) and Unique (frame-specific motion) components. By modeling these signals separately, COMUNI achieves state-of-the-art video quality while drastically reducing memory requirements and sampling time compared to traditional video diffusion models.
Introduction
The primary hurdle in high-resolution video generation is the "temporal tax." Unlike static images, videos possess an extra temporal dimension that exponentially increases the search space and computational complexity. Most existing models treat video frames as semi-independent entities, leading to significant redundancy. In typical video sequences, the background and the subject's primary appearance—the "common" signals—remain relatively stable, while only postures or expressions—the "unique" signals—evolve over time.
Modeling these disparate signals together is inherently inefficient. COMUNI proposes a clean separation of these components, allowing the generative model to focus on dynamic content without being bogged down by redundant pixel-level details.
Methodology: The COMUNI Framework
The COMUNI framework operates in two distinct stages: signal decomposition and latent diffusion generation.
Stage I: CU-VAE (Commonness and Uniqueness VAE)
Instead of a standard VAE that encodes frames individually, CU-VAE employs two specialized encoders:
- Commonness Encoder ($E_c$): Compresses an entire video clip into a single latent feature ($z_c$), capturing the global appearance.
- Uniqueness Encoder ($E_u$): Produces a sequence of features ($z_u^{1:T}$), one for each frame, representing motion and temporal changes.
To ensure effective separation, the authors implement an "implicit constraint" by keeping the spatial resolution of unique features small. This bottleneck forces the model to store high-detail appearance information in the common latent $z_c$. A Cascading Merge Module then recombines these features to reconstruct the original video during training.
Stage II: CU-LDM (Commonness and Uniqueness Latent Diffusion Model)
The generative engine utilizes two parallel diffusion streams to model $z_c$ and $z_u$ simultaneously. Joint Modules are interspersed between these streams, allowing for cross-communication to ensure that generated motion remains structurally consistent with the generated subject. Furthermore, a novel Position Embedding method incorporates absolute spatial and temporal information into the attention layers, maintaining structural integrity across the sequence.
Experiments and Results
COMUNI was evaluated on three major benchmarks: FaceForensics (facial expressions), UCF-101 (human actions), and Kinetics-400 (diverse real-world scenes).
Quantitative Performance
COMUNI sets new benchmarks in both quality and efficiency:
- Quality: On FaceForensics ($256^2$), it achieved a Fréchet Video Distance (FVD) of 55.2, outperforming the previous best of 62.5. On UCF-101 ($128^2$), it reached an Inception Score (IS) of 73.1, a new state-of-the-art.
- Efficiency: Generating 64 frames at $256^2$ resolution, COMUNI required only 110 seconds and 12GB of VRAM. In contrast, the VIDM baseline required 771 seconds and 20GB of VRAM, demonstrating COMUNI's viability for consumer-grade hardware.
Qualitative Comparison
Visual results demonstrate that COMUNI maintains better temporal consistency and higher image fidelity than competing models like DIGAN and StyleGAN-V.
Discussion and Limitations
While COMUNI represents a significant step forward, several challenges remain:
- Dynamic Backgrounds: In videos with rapid camera movement or chaotic backgrounds, the "common" signal is less distinct, which can lead to consistency issues.
- Temporal Repetition: Long-form generation can occasionally result in repetitive motion loops as the common feature remains static.
- Information Leakage: Despite the spatial bottleneck, unique features sometimes capture residual appearance information, which can cause artifacts during motion-transfer tasks.
Future research will focus on enhancing the robustness of the decomposition in highly dynamic environments and increasing motion diversity for extended video sequences.
Summary updated