Abstract
Video-to-video synthesis (vid2vid) aims for converting high-level semantic inputs to photorealistic videos. While existing vid2vid methods can achieve short-term temporal consistency, they fail to ensure the long-term one. This is because they lack knowledge of the 3D world being rendered and generate each frame only based on the past few frames. To address the limitation, we introduce a novel vid2vid framework that efficiently and effectively utilizes all past generated frames during rendering. This is achieved by condensing the 3D world rendered so far into a physically-grounded estimate of the current frame, which we call the guidance image. We further propose a novel neural network architecture to take advantage of the information stored in the guidance images. Extensive experimental results on several challenging datasets verify the effectiveness of our approach in achieving world consistency—the output video is consistent within the entire rendered 3D world.
Introduction
Video-to-video synthesis converts semantic inputs, such as segmentation masks or pose maps, into photorealistic video sequences. While previous state-of-the-art methods like the original vid2vid successfully maintain short-term temporal consistency (ensuring frame $t$ is similar to frame $t-1$), they frequently suffer from "long-term drift."
Because these models typically only consider the immediate past (the last 2-3 frames), they lack a concept of a persistent 3D world. For instance, if a virtual camera moves through a room and then returns to its starting point, the textures of the walls or the appearance of objects often change significantly. This lack of global coherence limits their use in applications like game engine rendering or long-form content creation.
Methodology: 3D-Grounded Synthesis
The proposed framework moves beyond 2D temporal warping by incorporating 3D geometry into the synthesis process. The core innovation is the use of guidance images—physically-grounded estimates of what the current frame should look like based on the entire history of the synthesis.
As shown in the pipeline above, the model "paints" generated colors and textures back onto a 3D point cloud of the scene, constructed via Structure from Motion (SfM). When the camera moves to a new position, this partially-textured point cloud is projected to create the guidance image.
Multi-SPADE Architecture
To utilize this information, the researchers developed a "Multi-SPADE" architecture. This module sequentially processes three types of data:
- Semantic Labels: Defining the scene structure.
- Warped Previous Frames: Maintaining short-term flow.
- Guidance Images: Enforcing long-term 3D consistency.
The model also employs partial convolutions to handle "holes" in guidance images—areas of the 3D world that have not yet been textured.
Experimental Results
The model was evaluated on datasets requiring high 3D structural integrity: Cityscapes (urban driving), MannequinChallenge (static people with moving cameras), and ScanNet (indoor sequences).
Key Findings
- World Consistency: As illustrated in the comparison above, the proposed method successfully "remembers" the appearance of a location. In a "round-trip" test (playing a video forward then in reverse), the method showed a significant reduction in pixel error compared to
vid2vid(e.g., 8.73 vs 14.90 RGB difference on Cityscapes). - Human Preference: In user studies, participants preferred the world-consistent results over baselines by a wide margin (0.73 vs 0.27 for realism on Cityscapes).
- Stereo Consistency: By leveraging a shared 3D point cloud, the model can generate consistent left-eye and right-eye views for VR applications, which is difficult for frame-by-frame generators.
Discussion and Future Work
While the method significantly improves long-term consistency, it remains dependent on the accuracy of the underlying Structure from Motion (SfM). Errors in 3D registration can lead to noisy guidance images. Additionally, the current model assumes a static world and does not account for dynamic lighting changes. Future research could explore 4D representations (3D + time) to handle complex, moving environments more robustly.
Summary updated