Accelerating the Training of Video Super-Resolution Models

Lijian Lin, Xintao Wang, Zhongang Qi, Ying Shan

Abstract

Despite that convolution neural networks (CNN) have recently demonstrated high-quality reconstruction for video super-resolution (VSR), efficiently training competitive VSR models remains a challenging problem. It usually takes an order of magnitude more time than training their counterpart image models, leading to long research cycles. Existing VSR methods typically train models with fixed spatial and temporal sizes from beginning to end. In this work, we show that it is possible to gradually train video models from small to large spatial/temporal sizes, i.e., in an easy-to-hard manner. Training is accelerated by such a multigrid training strategy, as most of computation is performed on smaller spatial and shorter temporal shapes. Extensive experiments demonstrate that our method is capable of largely speeding up training (up to $6.2\times$ speedup in wall-clock training time) without performance drop for various VSR models.

Introduction

Video Super-Resolution (VSR) aims to restore high-resolution (HR) video from low-resolution (LR) inputs by leveraging information across multiple frames. While effective, VSR models are computationally expensive to train. Because they must process both spatial (image height/width) and temporal (sequence length) dimensions, they often take an order of magnitude longer to train than their single-image counterparts.

Traditionally, researchers train these models using fixed, large spatial patches and long sequences from start to finish. This "rigid" strategy is inefficient: early in training, the model doesn't necessarily need high-resolution details to learn basic motion and structural features. The slow research cycle created by these long training times significantly hinders innovation in the field.

Multigrid Training Strategy

The authors propose a hierarchical "easy-to-hard" protocol to slash training times. Instead of fixed dimensions, the training is divided into two nested cycles:

  • Spatial Cycle: The training begins with small spatial patches (e.g., $32 \times 32$) and gradually moves to larger ones (e.g., $64 \times 64$).
  • Temporal Cycle: Inside each spatial stage, the model starts with short sequences (e.g., 6 frames) and gradually increases to the full sequence length (e.g., 15 frames).
Comparison between the standard baseline training and the proposed multigrid training strategy.

Crucially, they found that changing spatial and temporal sizes simultaneously is sub-optimal. Their hierarchical approach (Spatial $\rightarrow$ Temporal) ensures the model stabilizes on one dimension before the complexity of the other is increased.

Dynamic Learning Rate Scheduler

Switching to a harder task (larger patches/longer sequences) often causes an accuracy drop if the learning rate (LR) is already decayed. To fix this, the authors "re-start" the LR with a large value at the beginning of each new stage, using a cosine annealing strategy. This gives the model the "exploration power" needed to adapt to the increased complexity.

Large Minibatch Training

To maximize GPU utilization, the authors investigated training with much larger minibatches. They applied a linear scaling rule (increasing the LR proportionally with the batch size) and a warmup phase to ensure stability during the volatile early stages of training.

Key Results

The researchers evaluated their method on industry-standard VSR benchmarks like REDS and Vimeo-90K using popular architectures like BasicVSR and EDVR.

Training efficiency comparison: PSNR vs. wall-clock training time for BasicVSR-M.
  • Speedup: On the BasicVSR-M model, they achieved a 6.2× speedup in wall-clock training time.
  • Accuracy: Despite the massive reduction in training time, the final PSNR (Peak Signal-to-Noise Ratio) remained virtually identical to the baseline. For example, on REDS4, the baseline achieved 30.91 dB, while the accelerated "ours" version achieved 30.90 dB.
  • Generalization: The method proved effective across different types of models, including recurrent-based (BasicVSR) and sliding-window-based (EDVR) architectures.

Discussion and Limitations

The study highlights that hierarchical training is superior to synchronous scaling. Because spatial and temporal dimensions are highly correlated in video, the model learns more efficiently when these complexities are introduced sequentially. Most of the training iterations happen on small/short shapes where computation is cheap. By the time the model reaches the "hard" full-resolution stage, it has already learned the fundamental features.

However, some limitations exist:

  • Architecture Constraints: The speedup is less dramatic for sliding-window models like EDVR (2.3× vs 6.2×) because their temporal size is often fixed in the architecture.
  • Transformer Models: Future work is needed to see if these strategies translate to attention-based architectures, which have different computational bottlenecks.
  • Data Loading: As training becomes faster, the bottleneck may shift from the GPU to the CPU/Storage pipeline.

The code is available at https://github.com/TencentARC/Efficient-VSR-Training.

Summary updated

Abstract

Despite that convolution neural networks (CNN) have recently demonstrated high-quality reconstruction for video super-resolution (VSR), efficiently training competitive VSR models remains a challenging problem. It usually takes an order of magnitude more time than training their counterpart image models, leading to long research cycles. Existing VSR methods typically train models with fixed spatial and temporal sizes from beginning to end. The fixed sizes are usually set to large values for good performance, resulting to slow training. However, is such a rigid training strategy necessary for VSR? In this work, we show that it is possible to gradually train video models from small to large spatial/temporal sizes, i.e., in an easy-to-hard manner. In particular, the whole training is divided into several stages and the earlier stage has smaller training spatial shape. Inside each stage, the temporal size also varies from short to long while the spatial size remains unchanged. Training is accelerated by such a multigrid training strategy, as most of computation is performed on smaller spatial and shorter temporal shapes. For further acceleration with GPU parallelization, we also investigate the large minibatch training without the loss in accuracy. Extensive experiments demonstrate that our method is capable of largely speeding up training (up to $6.2\times$ speedup in wall-clock training time) without performance drop for various VSR models. The code is available at https://github.com/TencentARC/Efficient-VSR-Training.
BibTeX
@inproceedings{lin2023accelerating,
  author = {Lijian Lin and Xintao Wang and Zhongang Qi and Ying Shan},
  title = {{Accelerating the Training of Video Super-Resolution Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25246},
}
Figure