World-Consistent Video-to-Video Synthesis

Arun Mallya, Ting-Chun Wang, Karan Sapra, Ming-Yu Liu

Abstract

Video-to-video synthesis (vid2vid) aims for converting high-level semantic inputs to photorealistic videos. While existing vid2vid methods can achieve short-term temporal consistency, they fail to ensure the long-term one. This is because they lack knowledge of the 3D world being rendered and generate each frame only based on the past few frames. To address the limitation, we introduce a novel vid2vid framework that efficiently and effectively utilizes all past generated frames during rendering. This is achieved by condensing the 3D world rendered so far into a physically-grounded estimate of the current frame, which we call the guidance image. We further propose a novel neural network architecture to take advantage of the information stored in the guidance images. Extensive experimental results on several challenging datasets verify the effectiveness of our approach in achieving world consistency - the output video is consistent within the entire rendered 3D world.

Introduction: The Challenge of Long-Term Consistency

Video-to-video (vid2vid) synthesis is the task of transforming semantic maps—such as segmentation masks, sketches, or poses—into photorealistic video sequences. While current state-of-the-art models excel at generating high-quality individual frames and maintaining short-term smoothness (preventing flickering between adjacent frames), they suffer from a significant "memory" problem.

Standard vid2vid models typically generate each new frame based only on the immediate past (e.g., the last one or two frames). Consequently, they lack a global understanding of the 3D environment. If a virtual camera moves through a scene, turns away, and later returns to the same location, these models often generate entirely different textures or colors for the same objects. This "drift" breaks the illusion of a persistent 3D world.

Methodology: Anchoring Synthesis in 3D

To solve the drift problem, NVIDIA researchers developed a framework that provides the generative model with a 3D memory. The core innovation is the use of a Guidance Image to anchor the synthesis process in a consistent 3D world.

The Guidance Image Pipeline

The system operates by maintaining a persistent 3D representation of the scene generated so far:

  1. 3D Reconstruction: The framework utilizes Structure from Motion (SfM) to establish a 3D point cloud of the scene during training.
  2. Un-projection and Projection: As the video is synthesized, generated pixels are "painted" onto the 3D point cloud via un-projection. When the camera moves to a new pose, the model projects these painted 3D points back onto the 2D camera plane.
  3. Guidance as a Prior: This projection forms the "guidance image"—a physically-grounded estimate of what the scene should look like based on previous frames.
The guidance image generation pipeline using 3D point clouds.

Multi-SPADE Architecture

To effectively integrate this 3D information, the authors introduced the Multi-SPADE architecture. This module extends the Spatially-Adaptive Normalization (SPADE) technique by incorporating three distinct streams of information:

  • Semantic Labels: Defining the layout and object types.
  • Warped Previous Frames: Ensuring short-term temporal smoothness using standard optical flow.
  • Guidance Images: Providing the long-term 3D anchor for world consistency.

The model also employs Partial Convolutions to handle the inherent sparsity of guidance images, which often contain "holes" where 3D data is missing or hasn't been generated yet.

Experimental Evaluation

The framework was evaluated on three datasets requiring high 3D consistency: Cityscapes (urban driving), MannequinChallenge (complex camera movement around static people), and ScanNet (indoor environments).

Key Results and World Consistency

The primary metric for success was "world consistency"—the ability of the model to reproduce the same appearance when revisiting a location.

Overview of the input 3D source world and qualitative comparison of video consistency during a round-trip camera path.
  • Forward-Backward Consistency: In tests where a camera follows a path and then returns to its starting point, the proposed method reduced the visual error by 3x to 4x compared to previous baselines like vid2vid.
  • Human Preference: In blind studies, users preferred this method for both photorealism and temporal stability by a wide margin (typically over 70%).
  • Stereo Consistency: The 3D-aware nature of the model allows it to generate consistent views for stereo/VR applications, where independent frame generation typically fails to maintain binocular alignment.

Discussion and Future Directions

A key insight from this work is that temporal consistency is not merely a 2D warping problem but a 3D reconstruction problem. By treating video synthesis as the "just-in-time colorization" of a 3D world, the model gains a persistent memory that is robust even to noisy or sparse 3D data.

However, some limitations remain. The method currently relies on pre-computed 3D registrations (SfM) and assumes a largely static world. While optical flow handles dynamic objects, they do not yet benefit from the long-term 3D memory. Future research may explore disentangling albedo from shading to handle dynamic lighting or extending the 3D memory to dynamic scene elements.

Summary updated

Abstract

Video-to-video synthesis (vid2vid) aims for converting high-level semantic inputs to photorealistic videos. While existing vid2vid methods can achieve short-term temporal consistency, they fail to ensure the long-term one. This is because they lack knowledge of the 3D world being rendered and generate each frame only based on the past few frames. To address the limitation, we introduce a novel vid2vid framework that efficiently and effectively utilizes all past generated frames during rendering. This is achieved by condensing the 3D world rendered so far into a physically-grounded estimate of the current frame, which we call the guidance image. We further propose a novel neural network architecture to take advantage of the information stored in the guidance images. Extensive experimental results on several challenging datasets verify the effectiveness of our approach in achieving world consistency - the output video is consistent within the entire rendered 3D world. https://nvlabs.github.io/wc-vid2vid/
BibTeX
@inproceedings{mallya2020world,
  author = {Arun Mallya and Ting-Chun Wang and Karan Sapra and Ming-Yu Liu},
  title = {{World-Consistent Video-to-Video Synthesis}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=rvGB_g2IGG5},
}
Figure