AIM 2019 Challenge on Video Temporal Super-Resolution: Methods and Results

Seungjun Nah, Sanghyun Son, Radu Timofte, Kyoung Mu Lee

TL;DR

The AIM 2019 Challenge on Video Temporal Super-Resolution (VTSR) establishes a new benchmark for quadrupling video frame rates (from 15 to 60 fps) using the large-scale REDS_VTSR dataset. The challenge highlights a shift toward modeling non-linear motion and explores diverse architectures, ranging from high-performance optical flow models to innovative "zero-shot" learning approaches.

What Problem Does This Paper Address?

Video Temporal Super-Resolution—commonly known as frame interpolation—aims to synthesize missing frames between recorded ones to create smoother, higher-framerate video. This is critical for correcting the "stutter" seen in low-framerate footage caused by limited mobile processing power or low-light conditions.

The authors identify a major bottleneck in the field: the lack of a unified, high-quality benchmark. Existing datasets like Middlebury are too small, while others like Vimeo-90K lack the complex, non-linear motion found in real-world hand-held photography. The AIM 2019 challenge addresses this by introducing the REDS_VTSR dataset and providing a standardized environment to compare state-of-the-art deep learning solutions.

Methods

The challenge saw a variety of architectural strategies:

  • Quadratic Motion Modeling: The winning team, SenseSloMo, moved beyond the standard assumption that motion between frames is linear. By using four input frames to model acceleration (quadratic motion), they achieved superior results in dynamic scenes.
  • Flow-Free Interpolation: The NoFlow team challenged the status quo by avoiding explicit optical flow altogether. Instead, they used a deep encoder-decoder with channel attention to implicitly capture motion variations.
  • Depth-Aware Synthesis: Several teams, including DAIN and BOE_IOT_AIBD_IMP, integrated depth estimation to better handle occlusions, ensuring that closer objects correctly overlap distant ones during synthesis.
  • Zero-Shot Learning: The ZSFI team proposed a training-free approach that learns the internal statistics and patch recurrence of the specific test video itself to perform interpolation.

Datasets

The paper introduces REDS_VTSR, a dataset derived from diverse hand-held camera sequences. It consists of:

  • 300 sequences (240 training, 30 validation, 30 testing).
  • High Resolution: 1280 × 720 pixels.
  • Challenging Motion: Unlike synthetic datasets, these contain realistic camera shake and complex object dynamics.

The authors also provide a comprehensive comparison against older benchmarks such as UCF101, Sintel, and Adobe 240fps to contextualize the difficulty of the new task.

Example frames from various datasets compared to the new REDS_VTSR (h)

Key Insights and Ideas

  • Linearity is a Limitation: The success of SenseSloMo (Figure 4) demonstrates that for 4x upsampling (15 to 60 fps), assuming constant velocity is insufficient. Modeling the "physics" of motion (acceleration) is key to reducing ghosting artifacts.
  • Attention Can Replace Flow: The high ranking of the "NoFlow" method suggests that explicit motion vectors might not be strictly necessary if the network has enough capacity and the right attention mechanisms to resolve temporal correspondences.
  • Internal Statistics are Powerful: The ZSFI approach proves that a video often contains enough self-similarity across different temporal scales to interpolate itself, offering a path forward for specialized domains (like medical imaging) where massive training sets don't exist.

Key Results

The competition results (summarized in Figure 2 and Table 2) show a clear hierarchy of performance:

  • Top Performer: SenseSloMo led the pack with a PSNR of 25.47 dB and an SSIM of 0.7383 for 60 fps restoration.
  • Efficiency vs. Accuracy: While SenseSloMo was the most accurate, NoFlow offered a compelling trade-off, achieving second place (25.05 dB) with a significantly faster runtime (0.30s vs 1.00s per frame).
  • Qualitative Improvement: Top methods significantly reduced the "blurry overlay" effect seen in simpler baselines, producing much sharper edges on moving objects.
Qualitative comparison of the top-performing methods on a challenging sequence

Limitations and Future Work

  • Computational Overhead: Many of the top methods rely on heavy ensembles (e.g., flipping and rotating inputs multiple times) to squeeze out extra performance, making them difficult to deploy in real-time mobile applications.
  • Occlusion Handling: While depth-aware methods help, perfectly reconstructing pixels that are completely hidden in both reference frames remains an open challenge.
  • Artifact Suppression: The authors note that artifacts from high-speed recording itself can sometimes leak into the training data, suggesting a need for even cleaner ground-truth acquisition in the future.

Summary updated

Abstract

Videos contain various types and strengths of motions that may look unnaturally discontinuous in time when the recorded frame rate is low. This paper reviews the first AIM challenge on video temporal super-resolution (frame interpolation) with a focus on the proposed solutions and results. From low-frame-rate (15 fps) video sequences, the challenge participants are asked to submit higher-framerate (60 fps) video sequences by estimating temporally intermediate frames. We employ the REDS VTSR dataset derived from diverse videos captured in a hand-held camera for training and evaluation purposes. The competition had 62 registered participants, and a total of 8 teams competed in the final testing phase. The challenge winning methods achieve the state-of-the-art in video temporal superresolution.
BibTeX
@inproceedings{nah2019aim,
  author = {Seungjun Nah and Sanghyun Son and Radu Timofte and Kyoung Mu Lee},
  title = {{AIM 2019 Challenge on Video Temporal Super-Resolution: Methods and Results}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  doi = {10.1109/iccvw.2019.00421},
}
Figure