Abstract
Despite that convolution neural networks (CNN) have recently demonstrated high-quality reconstruction for video super-resolution (VSR), efficiently training competitive VSR models remains a challenging problem. It usually takes an order of magnitude more time than training their counterpart image models, leading to long research cycles. Existing VSR methods typically train models with fixed spatial and temporal sizes from beginning to end. In this work, we show that it is possible to gradually train video models from small to large spatial/temporal sizes, i.e., in an easy-to-hard manner. Training is accelerated by such a multigrid training strategy, as most of computation is performed on smaller spatial and shorter temporal shapes. Extensive experiments demonstrate that our method is capable of largely speeding up training (up to $6.2\times$ speedup in wall-clock training time) without performance drop for various VSR models.
Introduction
Video Super-Resolution (VSR) aims to restore high-resolution (HR) video from low-resolution (LR) inputs by leveraging information across multiple frames. While effective, VSR models are computationally expensive to train. Because they must process both spatial (image height/width) and temporal (sequence length) dimensions, they often take an order of magnitude longer to train than their single-image counterparts.
Traditionally, researchers train these models using fixed, large spatial patches and long sequences from start to finish. This "rigid" strategy is inefficient: early in training, the model doesn't necessarily need high-resolution details to learn basic motion and structural features. The slow research cycle created by these long training times significantly hinders innovation in the field.
Multigrid Training Strategy
The authors propose a hierarchical "easy-to-hard" protocol to slash training times. Instead of fixed dimensions, the training is divided into two nested cycles:
- Spatial Cycle: The training begins with small spatial patches (e.g., $32 \times 32$) and gradually moves to larger ones (e.g., $64 \times 64$).
- Temporal Cycle: Inside each spatial stage, the model starts with short sequences (e.g., 6 frames) and gradually increases to the full sequence length (e.g., 15 frames).
Crucially, they found that changing spatial and temporal sizes simultaneously is sub-optimal. Their hierarchical approach (Spatial $\rightarrow$ Temporal) ensures the model stabilizes on one dimension before the complexity of the other is increased.
Dynamic Learning Rate Scheduler
Switching to a harder task (larger patches/longer sequences) often causes an accuracy drop if the learning rate (LR) is already decayed. To fix this, the authors "re-start" the LR with a large value at the beginning of each new stage, using a cosine annealing strategy. This gives the model the "exploration power" needed to adapt to the increased complexity.
Large Minibatch Training
To maximize GPU utilization, the authors investigated training with much larger minibatches. They applied a linear scaling rule (increasing the LR proportionally with the batch size) and a warmup phase to ensure stability during the volatile early stages of training.
Key Results
The researchers evaluated their method on industry-standard VSR benchmarks like REDS and Vimeo-90K using popular architectures like BasicVSR and EDVR.
- Speedup: On the BasicVSR-M model, they achieved a 6.2× speedup in wall-clock training time.
- Accuracy: Despite the massive reduction in training time, the final PSNR (Peak Signal-to-Noise Ratio) remained virtually identical to the baseline. For example, on REDS4, the baseline achieved 30.91 dB, while the accelerated "ours" version achieved 30.90 dB.
- Generalization: The method proved effective across different types of models, including recurrent-based (BasicVSR) and sliding-window-based (EDVR) architectures.
Discussion and Limitations
The study highlights that hierarchical training is superior to synchronous scaling. Because spatial and temporal dimensions are highly correlated in video, the model learns more efficiently when these complexities are introduced sequentially. Most of the training iterations happen on small/short shapes where computation is cheap. By the time the model reaches the "hard" full-resolution stage, it has already learned the fundamental features.
However, some limitations exist:
- Architecture Constraints: The speedup is less dramatic for sliding-window models like EDVR (2.3× vs 6.2×) because their temporal size is often fixed in the architecture.
- Transformer Models: Future work is needed to see if these strategies translate to attention-based architectures, which have different computational bottlenecks.
- Data Loading: As training becomes faster, the bottleneck may shift from the GPU to the CPU/Storage pipeline.
The code is available at https://github.com/TencentARC/Efficient-VSR-Training.
Summary updated