Learning Temporally and Semantically Consistent Unpaired Video-to-Video Translation through Pseudo-Supervision from Synthetic Optical Flow

Kaihong Wang, Kumar Akash, Teruhisa MisuAAAI 2022 · DOI

Introduction

Unpaired video-to-video (V2V) translation is the challenging task of converting video content from one domain (e.g., a sunny day) to another (e.g., a rainy night) without matching "ground truth" pairs for training. While image-to-image translation has seen significant success, extending these techniques to video introduces a major hurdle: temporal inconsistency. When frames are translated independently, subtle pixel-level variations between adjacent frames are magnified, resulting in a distracting "flickering" effect and semantic drifting where objects lose their identity over time.

The Motion Estimation "Catch-22"

Previous state-of-the-art methods, such as Recycle-GAN, attempt to solve this by enforcing temporal consistency through estimated motion (optical flow). However, these methods suffer from a fundamental "Catch-22": motion estimation is inherently difficult, especially across different domains. If a motion estimator makes even a minor error, the consistency constraint forces the generator to produce blurry or incorrect pixels to match that mistake. This creates a cycle of instability where poor motion estimation directly degrades video quality.

Method: Synthetic Optical Flow for Pseudo-Supervision

To break this cycle, this paper introduces a paradigm shift from motion estimation to motion synthesis. Instead of guessing how objects move between two real frames, the authors apply a known synthetic optical flow ($\tilde{f}$) to a single static frame ($x_t$). This generates a "fake" future frame ($\tilde{x}_{t+1}$) with mathematically perfect ground truth for the motion.

Comparison of temporal consistency constraints: (a) Recycle-GAN, (b) standard optical flow-based methods, and (c) the proposed synthetic flow approach.

This "perfect" synthetic motion allows for the implementation of two novel losses that provide robust pseudo-supervision:

  1. Unsupervised Recycle Loss ($L_{ur}$): Ensures that translating a frame to the target domain, applying the synthetic warp, and translating it back recovers the synthetic future frame in the original domain.
  2. Unsupervised Spatial Loss ($L_{us}$): Enforces "commutativity"—the result of translating a frame and then warping it should be identical to warping the frame first and then translating it.

By utilizing these losses, the generator learns to respect motion dynamics without being misled by the errors of a pre-trained flow estimator.

Key Insights

  • The "Flawless" Supervisor: The synthetic flow does not need to be realistic to be effective. Even simple random grid warps provide a consistent, error-free signal that teaches the model how to handle temporal changes.
  • Breaking the Error Loop: By removing dependency on external estimators like FlowNet2 or RAFT, the model avoids inheriting the "domain gap" issues those estimators face when processing translated, potentially noisy images.
  • Efficiency: Synthetic flow is generated on the fly using simple mathematical operations, making the training process more computationally efficient than methods requiring heavy inference from external motion models.

Experimental Results

The proposed method achieved state-of-the-art results across several challenging benchmarks, including the Viper driving simulator and Cityscapes urban dataset.

  • Temporal Stability: In Viper-to-Cityscapes tasks, the "Warping Error" (measuring flickering) was reduced to 0.0359, significantly outperforming Recycle-GAN (0.0527) and nearing the stability of the original source video (0.0295).
  • Semantic Accuracy: The model showed a 28.96% improvement in mean Intersection over Union (mIoU) over Recycle-GAN in video-to-label translation, demonstrating superior consistency in maintaining object identities.
  • Human Preference: User studies on Amazon Mechanical Turk revealed that participants preferred this method over baselines in more than 75% of cases.

Conclusion and Future Work

By bypassing the limitations of motion estimation through synthetic pseudo-supervision, this approach provides a robust framework for temporally stable V2V translation. While highly effective for rigid motions typical of driving scenes, future work may explore more sophisticated synthetic flow generators to handle complex non-rigid motions, such as flowing water or hair, and extending consistency over longer video sequences.

Summary updated

BibTeX
@inproceedings{wang2022learning,
  author = {Kaihong Wang and Kumar Akash and Teruhisa Misu},
  title = {{Learning Temporally and Semantically Consistent Unpaired Video-to-Video Translation through Pseudo-Supervision from Synthetic Optical Flow}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i3.20148},
}
Figure