ReCoNet: Real-Time Coherent Video Style Transfer Network

Chang Gao, Derun Gu, Fangjun Zhang, Yizhou Yu

Introduction

Video style transfer aims to apply the artistic style of a reference image to a target video sequence. While image-based style transfer has seen significant success, extending these techniques to video introduces the challenge of temporal inconsistency. When processed frame-by-frame, slight variations in stylization between consecutive frames manifest as distracting "flickering" artifacts. This phenomenon occurs because independent frame processing fails to account for the temporal continuity of objects and lighting. ReCoNet addresses this by introducing a real-time, feed-forward network optimized for temporal coherence without sacrificing artistic quality or processing speed.

Methodology

ReCoNet utilizes a pure CNN-based encoder-decoder architecture designed for high-speed inference. Unlike optimization-based methods that require minutes per frame, ReCoNet's feed-forward design allows for massive parallelism on modern GPUs. The core innovation lies in its training objective, which incorporates multi-level temporal losses to ensure consistency across frames.

Multi-Level Temporal Losses

The network is trained using a "two-frame synergic" strategy, processing pairs of consecutive frames ($t-1$ and $t$) to calculate the following losses:

  1. Feature-map-level Temporal Loss: The authors argue that an object should have a consistent representation within the network's internal feature space. By penalizing differences in high-level feature maps between frames, the model learns to maintain a consistent stylistic "identity" for objects as they move through the scene.
  2. Output-level Temporal Loss with Luminance Warping: Traditional models often fail when lighting conditions change because they assume "brightness constancy." ReCoNet challenges this by incorporating a luminance warping constraint. This forces the stylized output to mirror the specific brightness changes present in the original input video, significantly improving stability under varying light.
  3. Training Data: To learn these temporal relationships, the model was trained on the Monkaa and FlyingThings3D datasets, which provide ground-truth optical flow and occlusion masks.

Importantly, the optical flow computation is only required during the training phase. During inference, ReCoNet operates as a pure feed-forward network, processing each frame independently and achieving speeds exceeding 200 FPS.

Key Results and Evaluation

ReCoNet demonstrates a significant leap in both performance and visual quality. On an NVIDIA GTX 1080 Ti, the model processes $640 \times 360$ video at 235.3 FPS, making it nearly 10x faster than previous real-time models.

Comparison of temporal coherence. Row (a) shows the style and input frames. Row (b) illustrates the flickering artifacts (green circles) in a baseline method. Row (c) shows ReCoNet's stable results with consistent textures across frames.

Quantitative and Qualitative Performance

Quantitative evaluations on the MPI Sintel dataset confirm that ReCoNet achieves significantly lower temporal error ($e_{stab}$) compared to previous state-of-the-art methods like Chen et al. Beyond metrics, a user study involving 50 participants found that ReCoNet was overwhelmingly preferred for its temporal consistency and overall stylistic quality. Participants noted that ReCoNet preserved rich artistic textures while maintaining a stable, flicker-free viewing experience even in complex, real-world scenes from the Videvo.net dataset.

Conclusion

By challenging the assumption of brightness constancy and enforcing stability at both the feature and output levels, ReCoNet provides a robust solution for real-time video style transfer. While the current model handles luminance changes effectively, future work will explore incorporating chromaticity (color) changes into the warping constraints to further improve stability in scenes with dramatic lighting shifts or complex color environments.

Summary updated

Abstract

Image style transfer models based on convolutional neural networks usually suffer from high temporal inconsistency when applied to videos. Some video style transfer models have been proposed to improve temporal consistency, yet they fail to guarantee fast processing speed, nice perceptual style quality and high temporal consistency at the same time. In this paper, we propose a novel real-time video style transfer model, ReCoNet, which can generate temporally coherent style transfer videos while maintaining favorable perceptual styles. A novel luminance warping constraint is added to the temporal loss at the output level to capture luminance changes between consecutive frames and increase stylization stability under illumination effects. We also propose a novel feature-map-level temporal loss to further enhance temporal consistency on traceable objects. Experimental results indicate that our model exhibits outstanding performance both qualitatively and quantitatively.
BibTeX
@inproceedings{gao2018reconet,
  author = {Chang Gao and Derun Gu and Fangjun Zhang and Yizhou Yu},
  title = {{ReCoNet: Real-Time Coherent Video Style Transfer Network}},
  booktitle = {ACCV},
  year = {2018},
  url = {https://openreview.net/forum?id=dw_4fjjxnW9M},
}
Figure