Abstract
Video Super-Resolution (VSR) models are traditionally difficult to train due to the high computational cost of processing three-dimensional data (spatial and temporal). Researchers from Tencent's ARC Lab have developed a "multigrid" training strategy that accelerates VSR model training by up to 6.2×. By employing an "easy-to-hard" curriculum that gradually increases spatial and temporal complexity and optimizing minibatch sizes, the proposed method achieves baseline-level accuracy in a fraction of the wall-clock time.
Introduction
Video Super-Resolution (VSR)—the task of reconstructing high-resolution video from low-resolution input—is notoriously computationally expensive. While image super-resolution deals with two dimensions (height and width), VSR adds a third: time. Currently, VSR models are trained using fixed, large spatial patches and long temporal sequences from start to finish. This "rigid" strategy leads to incredibly long research cycles; a single model can take weeks to train. This bottleneck slows down innovation in the field. The authors ask a fundamental question: Is it necessary to train on the hardest version of the problem from the very first iteration?
Methodology
The authors propose a two-pronged approach to slash training times:
1. Multigrid Training Strategy
Instead of fixed dimensions, the authors use a hierarchical "easy-to-hard" curriculum. This is divided into two primary cycles:
- Spatial Cycle: Training begins with small spatial resolutions (e.g., $32 \times 32$) and gradually moves to larger patches (e.g., $64 \times 64$).
- Temporal Cycle: Within each spatial stage, the number of consecutive frames (temporal length) also grows (e.g., from 7 to 15 frames).
To prevent accuracy drops when changing input sizes, the authors utilize a Dynamic Learning Rate Scheduler. They "re-start" the learning rate at a high value whenever the spatial or temporal size increases, allowing the model to adapt to the new, harder task.
2. Large Minibatch Training
To maximize GPU utilization, the authors investigated training with much larger minibatches. They found that by applying a linear scaling rule (increasing the learning rate proportionally to the batch size) and a warmup phase, they could process more samples per iteration without the typical accuracy penalty associated with large-batch training.
Key Insights
- The "Easy-to-Hard" Synergy: Training on small patches and short sequences early on allows the model to learn basic upsampling and alignment quickly. Once the model performs well on these simple tasks, it is better prepared to focus on the finer details required by larger, more complex inputs.
- Avoid Simultaneous Changes: Varying spatial and temporal sizes at the same time is counterproductive. Because spatial and temporal dimensions are highly correlated in video, changing both simultaneously confuses the learning process. A hierarchical approach (Spatial $\rightarrow$ Temporal) is key.
- The LR "Kickstart": Standard learning rate decay makes the learning rate very small toward the end of a cycle. Re-starting the learning rate provides the necessary flexibility for the model to adapt when task difficulty increases.
Experimental Results
The authors tested their method on popular VSR architectures including BasicVSR-M, BasicVSR, and EDVR-M using the REDS and Vimeo-90K datasets.
- Speedup: On the BasicVSR-M model, they achieved a 6.2× speedup in wall-clock training time. For the more complex BasicVSR, they saw a 3.1× speedup.
- Accuracy Maintenance: Despite the massive reduction in training time, the performance remained virtually identical to the baseline. For example, on REDS4, the accelerated BasicVSR-M achieved a PSNR of 30.90 dB compared to the baseline's 30.91 dB.
Conclusion
While highly effective, the multigrid approach has some constraints, particularly with "sliding-window" models like EDVR where temporal size is often fixed by the architecture. However, for recurrent models, the speedup is transformative. The authors suggest that this multigrid approach is a general strategy that could be extended to other video-based tasks such as deblurring or frame interpolation.
Summary updated