GAN Inversion for Consistent Video Interpolation and Manipulation

Weihao Xia, Yujiu Yang, Jing-Hao Xue

TL;DR

The authors introduce DynODE, a method that treats the latent space of a GAN as a high-dimensional dynamical system. By using Neural Ordinary Differential Equations (Neural ODEs) to model the trajectory of latent codes, they achieve smooth, infinite video interpolation and consistent video editing by manipulating only the very first frame.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs), particularly StyleGAN, are masters of high-resolution image synthesis. However, extending this power to video is notoriously difficult. Existing "video GANs" often struggle to match the quality of their single-image counterparts.

The current workaround is GAN inversion: taking each frame of a video and finding its corresponding "code" in the GAN's latent space. The problem? Most methods do this frame-by-frame. This approach is computationally expensive, redundant, and often fails to maintain temporal consistency—leading to "flickering" or identity shifts. The authors ask a pivotal question: instead of treating frames as isolated points, can we model the path (trajectory) these points take through the latent space?

Methods

The core innovation is viewing latent codes not as static vectors, but as moving particles in a dynamical system.

  1. Latent Trajectories: Given a sequence of inverted latent codes $\{z_0, z_1, \dots, z_n\}$, the authors assume these are discrete observations of a continuous function $z(t)$.
  2. Neural ODEs: They use a Neural ODE to learn the derivative function $f_\theta(z(t), t)$. Once trained, an ODE solver can predict the latent state at any time $t$ by starting from the initial frame $z_0$.
  3. Dynamic Space Mapping: The original latent space is effectively mapped into a "dynamic space" where transitions are modeled as continuous state changes.
  4. Consistent Manipulation: Because the entire video dynamics are tied to the first frame ($z_0$) and the learned trajectory, you can apply an edit (like adding a beard or changing the weather) to $z_0$ and simply "propagate" that change through the ODE. This ensures the edit follows the natural motion of the original video.

Datasets

The researchers evaluated DynODE across four distinct domains to prove its versatility:

  • Face: StyleGAN2 trained on LLHQ (1024×1024 resolution) using talking-head videos.
  • Scene: StyleGAN2 trained on Places365 (256×256) for outdoor natural scenes.
  • Bird: StyleGAN2 trained on the CUB-200-2011 dataset (256×256).
  • Isaac3D: A synthetic dataset (128×128) used to test controlled factors of variation like robot movement and camera height.

Key Insights and Ideas

  • Continuous vs. Discrete: Unlike Recurrent Neural Networks (RNNs), which require fixed time steps, Neural ODEs define a continuous latent state. This allows for infinite frame interpolation—you can generate a frame at $t=0.5$ or $t=0.001$ just as easily as $t=1$.
  • Redundancy Elimination: Traditional video editing requires applying the same GAN manipulation to every single frame. DynODE proves that if you model the dynamics correctly, you only need to edit the first frame. The "motion" is already captured by the ODE.
  • Holistic Geometry: The model doesn't just remember the frames it saw; it learns the "geometry" of the video's movement. This makes it robust to irregularly sampled video data.

Key Results

  • High Fidelity: On the Face dataset, the model achieved an impressive SSIM of 98.9% for observed frames and 93.2% for unobserved (interpolated) frames.
  • Temporal Consistency: Using identity preservation metrics (TL-ID and TG-ID), DynODE outperformed or matched state-of-the-art frame-by-frame editing methods while being significantly more computationally efficient.
  • Resolution Milestone: The authors successfully applied this to 1024×1024 videos, a substantial improvement over previous Neural ODE-based video generators that were often limited to much lower resolutions.

Limitations and Future Work

While powerful, the authors note several areas for growth:

  • Deterministic Nature: The current model is deterministic. Real-world video is often stochastic (unpredictable). The authors suggest that incorporating Stochastic Differential Equations (SDEs) could help model more complex, random motions.
  • Simplicity of the ODE: This work uses a relatively simple first-order ODE. More complex variants, like Neural Controlled Differential Equations (CDEs), might allow the model to adjust the trajectory better based on future observations.
  • Category Dependency: The method relies on a high-quality pretrained GAN for the specific category (e.g., faces, birds). If a good GAN doesn't exist for a specific niche, the method cannot be applied.

Reference: Xia, W., Yang, Y., & Xue, J. H. (2022). Modelling Latent Dynamics of StyleGAN using Neural ODEs. arXiv:2208.11197v2.

Summary updated

Abstract

In this paper, we propose to model the video dynamics by learning the trajectory of independently inverted latent codes from GANs. The entire sequence is seen as discrete-time observations of a continuous trajectory of the initial latent code, by considering each latent code as a moving particle and the latent space as a high-dimensional dynamic system. The latent codes representing different frames are therefore reformulated as state transitions of the initial frame, which can be modeled by neural ordinary differential equations. The learned continuous trajectory allows us to perform infinite frame interpolation and consistent video manipulation. The latter task is reintroduced for video editing with the advantage of requiring the core operations to be applied to the first frame only while maintaining temporal consistency across all frames. Extensive experiments demonstrate that our method achieves state-of-the-art performance but with much less computation. Code is available at https://github.com/weihaox/dynode_released.
BibTeX
@misc{xia2022gan,
  author = {Weihao Xia and Yujiu Yang and Jing-Hao Xue},
  title = {{GAN Inversion for Consistent Video Interpolation and Manipulation}},
  year = {2022},
  eprint = {2208.11197},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.11197},
}
Figure