MultiBoot Vsr: Multi-Stage Multi-Reference Bootstrapping for Video Super-Resolution.

Ratheesh Kalarot, Fatih PorikliWorkshops @ CVPR 2019 · DOI · OpenReview

TL;DR

This paper introduces MultiBoot VSR, a novel multi-stage, fully convolutional neural network for 4x video super-resolution. It leverages temporal redundancy by using multiple motion-compensated reference frames and a unique bootstrapping mechanism where the output of one stage becomes input for the next, progressively enhancing image quality. This approach achieves state-of-the-art results, ranking second in the NTIRE2019 VSR Challenge.

What Problem Does This Paper Address?

Video super-resolution (VSR) aims to reconstruct high-resolution (HR) video frames from low-resolution (LR) inputs. This is an inherently ill-posed problem because HR information is lost during downsampling. While single image super-resolution (SISR) methods exist, VSR can significantly benefit from the temporal redundancy present in video sequences. Existing VSR methods often struggle with temporal consistency, computational cost, and effectively utilizing multiple frames. Specifically, many deep learning approaches either require extensive training data, struggle with large motion, or have limitations in capturing global context due to patch-based processing. The authors aim to address these challenges by developing a VSR method that is class-agnostic, temporally consistent, and computationally efficient, while effectively exploiting multi-frame information.

Methods

The proposed MultiBoot VSR method is a multi-stage, fully convolutional neural network designed for 4x video super-resolution. Each stage of the network comprises three main components:

  1. Input Subnetwork: This component takes multiple motion-compensated reference frames (including the current frame) as input. It utilizes optical flow (estimated by the pre-trained FlowNet 2.0) to warp these reference frames to align with the current frame. The warped frames are then concatenated into a multi-channel tensor. For the first stage, this tensor has 15 channels (5 frames x 3 RGB channels). For subsequent stages, it can have more channels by incorporating rearranged HR outputs from previous stages.
  2. Blending Backbone: This is a deep, fully convolutional network composed of 16 residual blocks. It processes the LR feature maps generated by the input subnetwork. A direct skip connection from the input to the last residual block helps preserve original reference frame information and allows the network to learn the residual details needed for HR reconstruction. This deep structure with residual blocks and skip connections enhances the network's capacity and receptive field, enabling better contextual feedback.
  3. Spatial Upsampling Subnetwork: This subnetwork reconstructs the HR image from the feature maps produced by the blending backbone. It uses a pixel shuffling (space-to-depth) operation with learned weights, avoiding computationally expensive deconvolution layers. Each stage of the network performs a 2x super-resolution.

The core innovation is the multi-stage bootstrapping mechanism. After the first stage reconstructs HR frames, these outputs are rearranged (space-to-depth) into multiple LR images. These rearranged HR images, along with the original LR reference frames, are then fed into the next stage. This iterative process allows the network to progressively refine and enhance the image quality, leveraging its own previous estimations. The method is class-agnostic as it makes no assumptions about the objects in the images.

Datasets

The authors trained and evaluated their method using the REDS datasets provided by the NTIRE 2019 benchmark. These datasets contain HR videos (1280x720) and their corresponding LR versions (320x180), with 240 videos for training, 30 for validation, and 30 for testing. For training, they sampled approximately 1 million patches of size 128x128 from the training videos. For validation, they used 3000 patches of size 16 per image. Data augmentation techniques like rotation and flipping were applied.

Key Insights and Ideas

  • Bootstrapping for Progressive Enhancement: The multi-stage, recurrent application of the network, where previous HR outputs are fed back as LR inputs for subsequent stages, is a key idea. This allows the model to learn from its own improved estimations, leading to a progressive enhancement of image quality and temporal consistency.
  • Class-Agnostic Approach: By not making assumptions about object classes, the method is more generalizable and can be applied to a wider variety of video content without requiring class-specific training.
  • Efficient LR Processing: The network operates on LR feature maps throughout its blending backbone, significantly reducing memory requirements compared to methods that upsample early. This allows for deeper networks and larger receptive fields.
  • Pixel Shuffling for Upsampling: Using pixel shuffling instead of deconvolution for upsampling is computationally more efficient and avoids potential artifacts associated with deconvolution.
  • Huber Loss for Stability: The use of the Huber loss function is highlighted as crucial for stable training, combining the benefits of MAE (less sensitive to outliers) and MSE (differentiable and efficient for learning).

Key Results

  • Performance Ranking: The MultiBoot VSR method achieved the second-best SSIM score on the NTIRE 2019 VSR Challenge (Clean Track).
  • PSNR Improvement: After the challenge, fine-tuning the hyperparameters led to a further 0.3 dB improvement in PSNR.
  • Temporal Consistency and Artifact Reduction: The method generates temporally consistent results and sharp images without significant artifacts, even in challenging scenarios with fast motion and varying lighting.
  • Efficiency: The network runs at approximately 647 milliseconds for 32 residual blocks (for 4x SR), with optical flow computation taking an additional 200 milliseconds per image. While not fully optimized for GPU, this indicates a reasonable runtime.
  • Model Size vs. Performance: The paper notes that their network size (32 layers, 256-channel feature maps, 60M parameters) is not at the diminishing return point, suggesting room for further improvement with larger models.

Limitations and Future Work

  • Optical Flow Cost: The optical flow computation is a significant bottleneck, taking around 400 milliseconds per image. While the authors mention this can be reduced with GPU optimization, it remains a substantial part of the inference time.
  • Batch Size Limitations: Due to memory constraints with large tensors, the maximum batch size for training drops to 8 from 32 when using multiple stages, which can negatively affect convergence.
  • Future Work: The authors aim to investigate joint architectures that share weights and combine different processing branches. They also plan to train parts of the network with image super-resolution datasets and then fine-tune with video super-resolution datasets. They also suggest exploring more sophisticated data augmentation schemes with better computational resources.

Summary updated

Abstract

To make the best use of the previous estimations and shared redundancy across the consecutive video frames, here we propose a scene and class agnostic, fully convolutional neural network model for 4xvideo super-resolution. One stage of our network is composed of a motion compensation based input subnetwork, a blending backbone, and a spatial upsampling subnetwork. We recurrently apply this network to reconstruct high-resolution frames and then reuse them as additional reference frames after reshuffling them into multiple low-resolution images. This allows us to bootstrap and enhance image quality progressively. Our experiments show that our method generates temporally consistent and high-quality results without artifacts. Our method is ranked as the second best based on the SSIM scores on the NTIRE2019 VSR Challenge, Clean Track.
BibTeX
@inproceedings{kalarot2019multiboot,
  author = {Ratheesh Kalarot and Fatih Porikli},
  title = {{MultiBoot Vsr: Multi-Stage Multi-Reference Bootstrapping for Video Super-Resolution.}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00258},
}
Figure