COMUNI: Decomposing Common and Unique Video Signals for Diffusion-based Video Generation

Mingzhen Sun, Weining Wang, Xinxin Zhu, Jing LiuarXiv 2024 · DOI · arXiv

Abstract

Generating high-fidelity video remains a significant challenge due to the high computational costs associated with the temporal dimension. We present COMUNI, a novel two-stage diffusion framework that addresses this by decomposing video signals into Common (static appearance) and Unique (frame-specific motion) components. By modeling these signals separately, COMUNI achieves state-of-the-art video quality while drastically reducing memory requirements and sampling time compared to traditional video diffusion models.

Introduction

The primary hurdle in high-resolution video generation is the "temporal tax." Unlike static images, videos possess an extra temporal dimension that exponentially increases the search space and computational complexity. Most existing models treat video frames as semi-independent entities, leading to significant redundancy. In typical video sequences, the background and the subject's primary appearance—the "common" signals—remain relatively stable, while only postures or expressions—the "unique" signals—evolve over time.

Modeling these disparate signals together is inherently inefficient. COMUNI proposes a clean separation of these components, allowing the generative model to focus on dynamic content without being bogged down by redundant pixel-level details.

Methodology: The COMUNI Framework

The COMUNI framework operates in two distinct stages: signal decomposition and latent diffusion generation.

The two-stage COMUNI architecture: Stage I (CU-VAE) decomposes video into common and unique latents, while Stage II (CU-LDM) generates these latents through parallel diffusion streams.

Stage I: CU-VAE (Commonness and Uniqueness VAE)

Instead of a standard VAE that encodes frames individually, CU-VAE employs two specialized encoders:

  • Commonness Encoder ($E_c$): Compresses an entire video clip into a single latent feature ($z_c$), capturing the global appearance.
  • Uniqueness Encoder ($E_u$): Produces a sequence of features ($z_u^{1:T}$), one for each frame, representing motion and temporal changes.

To ensure effective separation, the authors implement an "implicit constraint" by keeping the spatial resolution of unique features small. This bottleneck forces the model to store high-detail appearance information in the common latent $z_c$. A Cascading Merge Module then recombines these features to reconstruct the original video during training.

Stage II: CU-LDM (Commonness and Uniqueness Latent Diffusion Model)

The generative engine utilizes two parallel diffusion streams to model $z_c$ and $z_u$ simultaneously. Joint Modules are interspersed between these streams, allowing for cross-communication to ensure that generated motion remains structurally consistent with the generated subject. Furthermore, a novel Position Embedding method incorporates absolute spatial and temporal information into the attention layers, maintaining structural integrity across the sequence.

Experiments and Results

COMUNI was evaluated on three major benchmarks: FaceForensics (facial expressions), UCF-101 (human actions), and Kinetics-400 (diverse real-world scenes).

Quantitative Performance

COMUNI sets new benchmarks in both quality and efficiency:

  • Quality: On FaceForensics ($256^2$), it achieved a Fréchet Video Distance (FVD) of 55.2, outperforming the previous best of 62.5. On UCF-101 ($128^2$), it reached an Inception Score (IS) of 73.1, a new state-of-the-art.
  • Efficiency: Generating 64 frames at $256^2$ resolution, COMUNI required only 110 seconds and 12GB of VRAM. In contrast, the VIDM baseline required 771 seconds and 20GB of VRAM, demonstrating COMUNI's viability for consumer-grade hardware.

Qualitative Comparison

Visual results demonstrate that COMUNI maintains better temporal consistency and higher image fidelity than competing models like DIGAN and StyleGAN-V.

Qualitative comparison of COMUNI against state-of-the-art baselines on FaceForensics and UCF101 datasets. COMUNI exhibits superior structural consistency and detail.

Discussion and Limitations

While COMUNI represents a significant step forward, several challenges remain:

  1. Dynamic Backgrounds: In videos with rapid camera movement or chaotic backgrounds, the "common" signal is less distinct, which can lead to consistency issues.
  2. Temporal Repetition: Long-form generation can occasionally result in repetitive motion loops as the common feature remains static.
  3. Information Leakage: Despite the spatial bottleneck, unique features sometimes capture residual appearance information, which can cause artifacts during motion-transfer tasks.

Future research will focus on enhancing the robustness of the decomposition in highly dynamic environments and increasing motion diversity for extended video sequences.

Summary updated

Abstract

Since videos record objects moving coherently, adjacent video frames have commonness (similar object appearances) and uniqueness (slightly changed postures). To prevent redundant modeling of common video signals, we propose a novel diffusion-based framework, named COMUNI, which decomposes the COMmon and UNIque video signals to enable efficient video generation. Our approach separates the decomposition of video signals from the task of video generation, thus reducing the computation complexity of generative models. In particular, we introduce CU-VAE to decompose video signals and encode them into latent features. To train CU-VAE in a self-supervised manner, we employ a cascading merge module to reconstitute video signals and a time-agnostic video decoder to reconstruct video frames. Then we propose CU-LDM to model latent features for video generation, which adopts two specific diffusion streams to simultaneously model the common and unique latent features. We further utilize additional joint modules for cross modeling of the common and unique latent features, and a novel position embedding method to ensure the content consistency and motion coherence of generated videos. The position embedding method incorporates spatial and temporal absolute position information into the joint modules. Extensive experiments demonstrate the necessity of decomposing common and unique video signals for video generation and the effectiveness and efficiency of our proposed method.
BibTeX
@misc{sun2024comuni,
  author = {Mingzhen Sun and Weining Wang and Xinxin Zhu and Jing Liu},
  title = {{COMUNI: Decomposing Common and Unique Video Signals for Diffusion-based Video Generation}},
  year = {2024},
  eprint = {2410.01718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.01718},
}
Figure