HyperCon: Image-to-Video Model Transfer for Video-to-Video Translation Tasks

Ryan Szeto, Mostafa El-Khamy, Jungwon Lee, Jason J. CorsoWACV 2021

TL;DR

This paper introduces HyperCon, a novel method for transferring image-to-image translation models to video-to-video tasks without fine-tuning. By interpolating frames and aggregating translated outputs, HyperCon achieves temporally consistent video translations, outperforming existing methods in both style transfer and inpainting tasks.

What Problem Does This Paper Address?

Video-to-video translation, such as style transfer or inpainting on videos, presents a significant challenge: maintaining temporal consistency. While image-to-image translation models excel at capturing visual knowledge from vast datasets, directly applying them to videos often results in distracting flickering artifacts. Conversely, video-specific models can achieve temporal consistency but struggle to leverage the rich visual understanding learned from large image datasets, leading to poorer generalization. Training video models from scratch for every task is also computationally prohibitive due to the sheer volume of video data required. This paper aims to bridge this gap by enabling image-to-image models to perform video-to-video translation tasks with temporal consistency, without requiring task-specific video training.

Methods

HyperCon is an "image-to-video model transfer" method that works in three main stages:

  1. Frame Interpolation: The input video is augmented by inserting synthesized intermediate frames between existing ones. This is done using a vector-based sampling method with a frame interpolation network (specifically, Super SloMo [16]) that predicts warping grids and weight maps. This step increases the temporal density of the video. For masked inputs (like inpainting), an interpolated mask video is also generated.
  2. Frame-wise Translation: The pre-trained, black-box image-to-image translation model (e.g., a style transfer or inpainting network) is applied independently to each frame of the interpolated video. This stage leverages the generalization capabilities of the image model.
  3. Temporal Aggregation: The translated, interpolated frames are then aggregated over local sliding windows. To ensure consistency, frames within these windows are aligned using optical flow (estimated by PWC-Net [29]) to a reference frame. Pixel-wise pooling (mean or median) is then applied to produce the final, temporally consistent output video frames. This aggregation step is crucial for smoothing out the frame-wise translation artifacts and enforcing temporal coherence.

A key innovation is the handling of masked inputs, where HyperCon adapts by interpolating masks and ensuring the translation model receives both RGB frames and their corresponding masks.

Datasets

The paper evaluates HyperCon on several standard video datasets:

  • ActivityNet [3]: Used for video style transfer. Clips were manually curated from the official training/validation and test splits.
  • DAVIS [25]: Used for both video style transfer and video inpainting. The official training/validation set was used for validation, and development/challenge sets from DAVIS 2017 and 2019 were combined for test sets.
  • Places [42]: Used for evaluating the re-trained inpainting model's performance.

No new benchmarks were introduced; instead, existing datasets were used to demonstrate HyperCon's effectiveness on established video-to-video translation tasks.

Key Insights and Ideas

  • Leveraging Image Model Generalization: The core idea is that image models, trained on massive datasets, possess superior generalization capabilities. HyperCon provides a mechanism to transfer this generalization power to video tasks, overcoming the limitations of smaller, task-specific video datasets.
  • Interpolation as a Temporal Regularizer: Artificially increasing the frame rate through interpolation creates more temporal information. Aggregating over these denser frames allows for a more robust estimation of stable visual components, effectively smoothing out flickering.
  • Decoupling Translation and Consistency: HyperCon separates the image translation process from the temporal consistency enforcement. This allows for the use of any pre-trained image-to-image model without modification, making it highly versatile.
  • Unified Framework for Masked/Unmasked Tasks: The method's ability to handle both unmasked (style transfer) and masked (inpainting) inputs within a single framework is a significant advantage, broadening its applicability.

Key Results

  • Video Style Transfer: HyperCon significantly outperforms baseline methods like FST [17] and FST-vcons [19] in terms of temporal consistency (lower Ewarp and higher CPSNR/CSSIM scores) while maintaining comparable or better style adherence (lower FID). Human evaluations also favored HyperCon.
  • Video Inpainting: HyperCon models (using mean or median aggregation) consistently ranked among the top two methods across various metrics (Ewarp, DLPIPS, VLPIPS, FID, VFID) on DAVIS 2017, DAVIS 2019, and ActivityNet datasets. It notably outperforms VINet [18], a model specifically trained for video inpainting, despite HyperCon not being trained on masked videos. Qualitative results show HyperCon better recovers boundaries and textures.

Limitations and Future Work

  • Limited Long-Range Temporal Dependencies: HyperCon's aggregation mechanism is local. It does not explicitly enforce dependencies over very long video sequences, which could be a limitation for extremely long videos.
  • Propagation of Frame-wise Errors: As with other image-to-video transfer methods, HyperCon can propagate egregious errors from the underlying image-to-image model.
  • Future Work: The authors suggest integrating the three HyperCon steps into a single, trainable, parameter-efficient model. This could potentially improve performance by allowing for shared information between stages and enabling more efficient processing of longer videos.

Summary updated

BibTeX
@inproceedings{szeto2021hypercon,
  author = {Ryan Szeto and Mostafa El-Khamy and Jungwon Lee and Jason J. Corso},
  title = {{HyperCon: Image-to-Video Model Transfer for Video-to-Video Translation Tasks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure