Video-to-Video Translation with Global Temporal Consistency

Xingxing Wei, Jun Zhu, Sitong Feng, Hang Su

Abstract

Although image-to-image translation has been widely studied, the video-to-video translation is rarely mentioned. In this paper, we propose an unified video-to-video translation framework to accom- plish different tasks, like video super-resolution, video colouriza- tion, and video segmentation, etc. A consequent question within video-to-video translation lies in the flickering appearance along with the varying frames. To overcome this issue, a usual method is to incorporate the temporal loss between adjacent frames in the optimization, which is a kind of local frame-wise temporal con- sistency. We instead present a residual error based mechanism to ensure the video-level consistency of the same location in different frames (called (lobal temporal consistency). The global and local consistency are simultaneously integrated into our video-to-video framework to achieve more stable videos. Our method is based on the GAN framework, where we present a two-channel discrimina- tor. One channel is to encode the video RGB space, and another is to encode the residual error of the video as a whole to meet the global consistency. Extensive experiments conducted on different video- to-video translation tasks verify the effectiveness and flexibleness of the proposed method.
BibTeX
@inproceedings{wei2018video,
  author = {Xingxing Wei and Jun Zhu and Sitong Feng and Hang Su},
  title = {{Video-to-Video Translation with Global Temporal Consistency}},
  booktitle = {ACM Multimedia},
  year = {2018},
  doi = {10.1145/3240508.3240708},
}

Loading related content...

Figure