Accelerating the Training of Video Super-resolution Models

Lijian Lin, Xintao Wang, Zhongang Qi, Ying Shan

Abstract

Video Super-Resolution (VSR) models are traditionally difficult to train due to the high computational cost of processing three-dimensional data (spatial and temporal). Researchers from Tencent's ARC Lab have developed a "multigrid" training strategy that accelerates VSR model training by up to 6.2×. By employing an "easy-to-hard" curriculum that gradually increases spatial and temporal complexity and optimizing minibatch sizes, the proposed method achieves baseline-level accuracy in a fraction of the wall-clock time.

Introduction

Video Super-Resolution (VSR)—the task of reconstructing high-resolution video from low-resolution input—is notoriously computationally expensive. While image super-resolution deals with two dimensions (height and width), VSR adds a third: time. Currently, VSR models are trained using fixed, large spatial patches and long temporal sequences from start to finish. This "rigid" strategy leads to incredibly long research cycles; a single model can take weeks to train. This bottleneck slows down innovation in the field. The authors ask a fundamental question: Is it necessary to train on the hardest version of the problem from the very first iteration?

Methodology

The authors propose a two-pronged approach to slash training times:

1. Multigrid Training Strategy

Instead of fixed dimensions, the authors use a hierarchical "easy-to-hard" curriculum. This is divided into two primary cycles:

  • Spatial Cycle: Training begins with small spatial resolutions (e.g., $32 \times 32$) and gradually moves to larger patches (e.g., $64 \times 64$).
  • Temporal Cycle: Within each spatial stage, the number of consecutive frames (temporal length) also grows (e.g., from 7 to 15 frames).

To prevent accuracy drops when changing input sizes, the authors utilize a Dynamic Learning Rate Scheduler. They "re-start" the learning rate at a high value whenever the spatial or temporal size increases, allowing the model to adapt to the new, harder task.

Comparison of baseline training with fixed dimensions versus the proposed multigrid strategy using spatial and temporal cycles.

2. Large Minibatch Training

To maximize GPU utilization, the authors investigated training with much larger minibatches. They found that by applying a linear scaling rule (increasing the learning rate proportionally to the batch size) and a warmup phase, they could process more samples per iteration without the typical accuracy penalty associated with large-batch training.

Key Insights

  • The "Easy-to-Hard" Synergy: Training on small patches and short sequences early on allows the model to learn basic upsampling and alignment quickly. Once the model performs well on these simple tasks, it is better prepared to focus on the finer details required by larger, more complex inputs.
  • Avoid Simultaneous Changes: Varying spatial and temporal sizes at the same time is counterproductive. Because spatial and temporal dimensions are highly correlated in video, changing both simultaneously confuses the learning process. A hierarchical approach (Spatial $\rightarrow$ Temporal) is key.
  • The LR "Kickstart": Standard learning rate decay makes the learning rate very small toward the end of a cycle. Re-starting the learning rate provides the necessary flexibility for the model to adapt when task difficulty increases.

Experimental Results

The authors tested their method on popular VSR architectures including BasicVSR-M, BasicVSR, and EDVR-M using the REDS and Vimeo-90K datasets.

  • Speedup: On the BasicVSR-M model, they achieved a 6.2× speedup in wall-clock training time. For the more complex BasicVSR, they saw a 3.1× speedup.
  • Accuracy Maintenance: Despite the massive reduction in training time, the performance remained virtually identical to the baseline. For example, on REDS4, the accelerated BasicVSR-M achieved a PSNR of 30.90 dB compared to the baseline's 30.91 dB.
Training efficiency comparison: the proposed method achieves nearly identical PSNR to the baseline while being 6.2x faster.

Conclusion

While highly effective, the multigrid approach has some constraints, particularly with "sliding-window" models like EDVR where temporal size is often fixed by the architecture. However, for recurrent models, the speedup is transformative. The authors suggest that this multigrid approach is a general strategy that could be extended to other video-based tasks such as deblurring or frame interpolation.

Summary updated

Abstract

Despite that convolution neural networks (CNN) have recently demonstrated high-quality reconstruction for video super-resolution (VSR), efficiently training competitive VSR models remains a challenging problem. It usually takes an order of magnitude more time than training their counterpart image models, leading to long research cycles. Existing VSR methods typically train models with fixed spatial and temporal sizes from beginning to end. The fixed sizes are usually set to large values for good performance, resulting to slow training. However, is such a rigid training strategy necessary for VSR? In this work, we show that it is possible to gradually train video models from small to large spatial/temporal sizes, \ie, in an easy-to-hard manner. In particular, the whole training is divided into several stages and the earlier stage has smaller training spatial shape. Inside each stage, the temporal size also varies from short to long while the spatial size remains unchanged. Training is accelerated by such a multigrid training strategy, as most of computation is performed on smaller spatial and shorter temporal shapes. For further acceleration with GPU parallelization, we also investigate the large minibatch training without the loss in accuracy. Extensive experiments demonstrate that our method is capable of largely speeding up training (up to $6.2\times$ speedup in wall-clock training time) without performance drop for various VSR models.
BibTeX
@inproceedings{lin2023accelerating,
  author = {Lijian Lin and Xintao Wang and Zhongang Qi and Ying Shan},
  title = {{Accelerating the Training of Video Super-resolution Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25246},
}
Figure