Video Prediction with Appearance and Motion Conditions

Yunseok Jang, Gunhee Kim, Yale Song

TL;DR

The paper introduces AMC-GAN, a conditional generative adversarial network that predicts future video frames by explicitly disentangling appearance from motion. By using a single frame for appearance and auxiliary variables for motion, it generates high-fidelity, controllable videos that significantly outperform traditional extrapolation-based methods.

What Problem Does This Paper Address?

Predicting the future is inherently uncertain. If you see a person standing still, they could start walking, waving, or sitting down. Traditional video prediction models that try to "extrapolate" from a few past frames often struggle with this ambiguity, resulting in blurry, "averaged" images that represent a mix of all possible futures.

The authors aim to solve this "ill-posed" problem by providing the model with specific conditions—appearance and motion—to reduce uncertainty. The goal is to generate sharp, realistic videos where the specific action performed is controllable and visually distinct.

Methods

The core innovation is the Appearance-Motion Conditional GAN (AMC-GAN), which breaks down the prediction task into two controllable pathways:

  • Disentangled Inputs: Instead of learning from a sequence of frames where appearance and motion are entangled, the model takes a single "appearance" frame ($y_a$) and an auxiliary "motion" variable ($y_m$). This motion variable includes both high-level category labels and low-level keypoint velocities.
  • Dual Discriminators: The architecture employs two specialized discriminators. $D_a$ (Appearance) ensures that generated frames maintain the identity and visual consistency of the input frame. $D_m$ (Motion) ensures the sequence of movements is realistic and matches the requested action label.
  • Perceptual Triplet Ranking: A novel ranking module ($R$) uses a triplet loss to ensure that videos generated under similar conditions look more similar to each other than those generated under different conditions.
  • Mismatched Conditioning Scheme: To prevent the model from ignoring the input conditions (a common failure mode in GANs), the authors train the discriminators on "mismatched" pairs—such as a real video of a "happy" face paired with a "sad" label—forcing the generator to strictly adhere to the provided motion cues to fool the discriminator.
Figure 2. An overview of our AMC-GAN; we provide architecture and implementation details in the supplementary material.

Datasets

The model was evaluated on two distinct benchmarks:

  1. MUG Facial Expression Dataset: Contains 931 clips of six basic emotions (anger, disgust, fear, happy, sad, surprise). The authors used 11 facial landmarks as keypoints.
  2. NATOPS Human Action Dataset: A more complex set of 9,600 clips featuring 24 different aircraft handling signals. This requires coordinating body and hand movements, using 9 joint locations as keypoints.

Key Insights and Ideas

  • Disentanglement is Directing: By making motion an explicit input, the model becomes a "director." The same neutral starting face can be made to smile, frown, or look surprised simply by swapping the motion variable.
  • Single-Frame Superiority: The authors found that learning from a single frame plus explicit motion cues is more effective for generating sharp results than trying to extrapolate from multiple frames, which often introduces blur due to uncertainty.
  • The Power of Constraints: The combination of dual discriminators and triplet ranking acts as a powerful regularizer, ensuring that the model doesn't just generate any realistic video, but specifically the one requested by the input conditions.

Key Results

  • Massive Accuracy Gains: To test if the generated videos actually contained the intended motion, the authors ran a separate video classifier on the results. On the NATOPS dataset, AMC-GAN achieved 91.12% classification accuracy, compared to just 12.76% for the next best baseline (MCnet).
  • Human Preference: In subjective tests, humans preferred AMC-GAN's results over baselines by a wide margin. For the NATOPS dataset, participants preferred AMC-GAN over MCnet 98.2% of the time.
  • Visual Fidelity: Qualitatively, the model produces noticeably sharper frames and more distinct motion patterns than previous methods like CDNA, which often result in ghostly, blurry motion.
Visual results showing how a single appearance frame (ya) can be transformed into various expressions or actions based on different motion conditions (ym).

Limitations and Future Work

  • Motion Priors: The motion condition ($y_m$) requires keypoint velocities. While these are known during training, they must be sampled or estimated during inference, which might limit the diversity of generated movements to what has been seen in the training set.
  • Keypoint Dependency: The system relies on the quality of external keypoint detectors (like OpenFace). Errors in these detectors can propagate to the generated video.
  • Future Directions: The authors suggest exploring ways to discover these motion conditions in a purely unsupervised manner or applying the framework to even more complex, unstructured real-world scenes.

Summary updated

Abstract

Video prediction aims to generate realistic future frames by learning dynamic visual patterns. One fundamental challenge is to deal with future uncertainty: How should a model behave when there are multiple correct, equally probable future? We propose an Appearance-Motion Conditional GAN to address this challenge. We provide appearance and motion information as conditions that specify how the future may look like, reducing the level of uncertainty. Our model consists of a generator, two discriminators taking charge of appearance and motion pathways, and a perceptual ranking module that encourages videos of similar conditions to look similar. To train our model, we develop a novel conditioning scheme that consists of different combinations of appearance and motion conditions. We evaluate our model using facial expression and human action datasets and report favorable results compared to existing methods.
BibTeX
@inproceedings{jang2018video,
  author = {Yunseok Jang and Gunhee Kim and Yale Song},
  title = {{Video Prediction with Appearance and Motion Conditions}},
  booktitle = {International Conference on Machine Learning},
  year = {2018},
}
Figure