TwoStreamVAN: Improving Motion Modeling in Video Generation

Ximeng Sun, Huijuan Xu, Kate SaenkoWACV 2020 · OpenReview

Abstract

Video generation is an inherently challenging task, as it requires modeling realistic temporal dynamics as well as spatial content. Existing methods entangle the two intrinsically different tasks of motion and content creation in a single generator network, but this approach struggles to simultaneously generate plausible motion and content. To improve motion modeling in video generation task, we propose a two-stream model that disentangles motion generation from content generation, called a Two-Stream Variational Adversarial Network (TwoStreamVAN). Given an action label and a noise vector, our model is able to create clear and consistent motion, and thus yields photorealistic videos. The key idea is to progressively generate and fuse multi-scale motion with its corresponding spatial content. Our model significantly outperforms existing methods on the standard Weizmann Human Action, MUG Facial Expression and VoxCeleb datasets, as well as our new dataset of diverse human actions with challenging and complex motion.

Introduction

Generating realistic video requires choreographing a sequence where every pixel moves convincingly over time while maintaining its identity. Most existing video generation models utilize a single "entangled" generator that attempts to learn background, foreground objects, and complex physics simultaneously. This often results in visual artifacts: limbs disappearing into backgrounds, warping faces, or "wiggling" static backgrounds.

The core issue is that content (static structure) and motion (dynamic change) are intrinsically different. When lumped together, motion modeling often interferes with content quality, leading to blurry frames and unrealistic temporal dynamics. TwoStreamVAN addresses this by separating what an object looks like from how it moves.

Methodology

The Two-Stream Variational Adversarial Network (TwoStreamVAN) breaks the generation process into two specialized paths:

  1. The Content Stream: An action-conditioned VAE/GAN focused on generating high-quality static frames, learning the "identity" of the subject and environment.
  2. The Motion Stream: Utilizes a ConvLSTM to model temporal dynamics. Rather than generating pixels directly, it produces pixel-wise 2D motion kernels and motion masks.
The TwoStreamVAN architecture disentangles motion and content into parallel streams.

Multi-scale Fusion

The architecture implements a multi-scale fusion mechanism where the two streams interact at various resolutions. Low-resolution layers handle large, sweeping motions (e.g., jumping), while high-resolution layers manage fine-grained refinements (e.g., facial expressions). At each scale, the model predicts a motion mask. This mask identifies where motion occurs; where the mask is zero, the model preserves static pixels from the content stream, effectively "locking" the background and preventing artifacts.

Datasets and Evaluation

The model was evaluated on several benchmarks:

  • Weizmann Human Action: 90 videos of 10 actions.
  • MUG Facial Expression: 882 videos of 6 expressions.
  • VoxCeleb: Large-scale dataset of speaking individuals.
  • SynAction (New): A large-scale synthetic dataset created in Unity featuring 120 unique non-rigid human action models across 20 classes, including complex physics like running and squatting.
Generated video sequences from the SynAction dataset showing complex motions like squatting and running.

Results

TwoStreamVAN significantly outperformed baselines such as MoCoGAN and VideoVAE:

  • Quantitative Performance: On the Weizmann dataset, TwoStreamVAN achieved an Inception Score (IS) of 77.11, compared to MoCoGAN’s 58.52.
  • Qualitative Quality: Visual results demonstrate sharper outlines of moving limbs. In VoxCeleb tests, the model captured subtle lip and head movements accurately, avoiding the "mode collapse" seen in other models.
  • User Preference: In a blind study on the SynAction dataset, 88% of users preferred TwoStreamVAN-generated videos over competitors.

Conclusion

TwoStreamVAN demonstrates that disentangling motion and content within the generator architecture—rather than just the latent space—leads to superior video fidelity. By explicitly modeling where motion should not happen via masks and using kernels to maintain appearance consistency, the model achieves state-of-the-art results. Future work includes exploring text-to-video generation and interactive motion manipulation.

Code and the SynAction dataset are available at: https://github.com/sunxm2357/TwoStreamVAN/

Summary updated

Abstract

Video generation is an inherently challenging task, as it requires modeling realistic temporal dynamics as well as spatial content. Existing methods entangle the two intrinsically different tasks of motion and content creation in a single generator network, but this approach struggles to simultaneously generate plausible motion and content. To improve motion modeling in video generation task, we propose a two-stream model that disentangles motion generation from content generation, called a Two-Stream Variational Adversarial Network (TwoStreamVAN). Given an action label and a noise vector, our model is able to create clear and consistent motion, and thus yields photorealistic videos. The key idea is to progressively generate and fuse multi-scale motion with its corresponding spatial content. Our model significantly outperforms existing methods on the standard Weizmann Human Action, MUG Facial Expression and VoxCeleb datasets, as well as our new dataset of diverse human actions with challenging and complex motion. Our code is available at https://github.com/sunxm2357/TwoStreamVAN/.
BibTeX
@inproceedings{sun2020twostreamvan,
  author = {Ximeng Sun and Huijuan Xu and Kate Saenko},
  title = {{TwoStreamVAN: Improving Motion Modeling in Video Generation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=EdRYusKT8X},
}
Figure