Abstract
Video Super-Resolution (VSR) is a critical task in computer vision that involves restoring high-resolution (HR) frames from low-resolution (LR) sequences. Current methods often struggle to balance the need for long-range spatial-temporal modeling with the strict computational constraints of edge devices. This paper introduces MFPI, a novel VSR framework that achieves state-of-the-art performance through two key innovations: a Multi-Frequency Representation Enhancement (MFE) module and a Privilege Training (PT) strategy. By operating in the frequency domain, MFPI captures global dependencies more efficiently than Transformer-based architectures. Furthermore, the PT strategy allows the model to leverage HR information during training to optimize internal representations without incurring any additional cost during inference. Experimental results across multiple benchmarks demonstrate that MFPI outperforms existing methods in both accuracy and efficiency.
Introduction
The primary challenge in Video Super-Resolution lies in effectively utilizing information across both spatial and temporal dimensions. Traditional Convolutional Neural Networks (CNNs) are often limited by small, local receptive fields, which prevent them from capturing long-range spatial patterns or complex motions across frames. While Vision Transformers (ViTs) have emerged as a solution for global modeling, their high computational complexity ($O(N^2)$) makes them impractical for real-time applications on mobile or edge hardware.
MFPI addresses this gap by asking: Can we provide CNNs with the global "vision" of a Transformer without the massive computational overhead? The authors propose shifting certain operations to the frequency domain and utilizing a "cheat sheet" approach during training to guide the network toward more effective feature extraction.
Methodology
Multi-Frequency Representation Enhancement (MFE)
To overcome the limitations of local receptive fields, the MFE module processes video data in the frequency domain rather than solely in the spatial domain. This module is composed of three distinct branches designed to capture different types of information:
- Spatial-Frequency Enhancement (SFE): This branch utilizes Fast Fourier Transforms (FFT) and learnable filters. In the frequency domain, a single point represents global information from the spatial domain. This allows the model to achieve a "global receptive field" with a complexity of only $O(N \log N)$, significantly lower than the self-attention mechanisms used in Transformers.
- Energy-Frequency Enhancement (EFE): Using Discrete Cosine Transforms (DCT) and a specialized energy function, this branch models relationships between feature channels. This helps the network prioritize salient information and suppress the noise that often accumulates in multi-frame video sequences.
- Auxiliary Branch: A standard pair of depthwise convolutions is included to maintain the local inductive bias that CNNs are naturally proficient at.
Privilege Training (PT)
Inspired by the concept of "Learning Using Privilege Information" (LUPI), the authors introduce a novel training strategy. During the training phase, the model is allowed to "peek" at the ground-truth HR frames through a lightweight encoder. A mixing parameter ($\gamma$) controls the amount of HR information integrated into the LR input. As training progresses, $\gamma$ is gradually reduced to zero. This "training wheels" approach guides the model toward a better mathematical minima early in the process. Crucially, because $\gamma$ reaches zero by the end of training, the model requires zero extra information or parameters during actual inference.
Experimental Evaluation
Datasets
The performance of MFPI was rigorously evaluated on four standard VSR benchmarks:
- REDS4: Known for high-quality video and complex motion patterns.
- Vimeo-90K / Vimeo-90K-T: A large-scale dataset for general VSR tasks.
- Vid4: A classic benchmark for video restoration.
- UDM10: Specifically used to evaluate performance under different degradation types, including Bicubic and Blur Downsampling (BD).
Results and Discussion
MFPI consistently outperformed eighteen previous state-of-the-art methods across all tested benchmarks. On the UDM10 dataset (BD degradation), MFPI achieved a 0.36 dB PSNR improvement over BasicVSR++, the previous gold standard.
Beyond accuracy, MFPI demonstrates exceptional efficiency. With approximately 7.3M parameters and a runtime of 76ms for a standard 180x320 input, it is significantly lighter and faster than competing Transformer-based models. Visual analysis confirms that MFPI is particularly adept at recovering fine, repeating textures—such as text on signs or intricate patterns in clothing—that other models typically blur.
Conclusion and Future Work
MFPI represents a significant advancement in efficient Video Super-Resolution by combining frequency-domain global modeling with a privileged training strategy. While the model shows great promise on synthetic benchmarks, future work will focus on testing MFPI against real-world degradations and complex unknown noise. Additionally, the authors plan to explore the application of the MFE and PT components to other video restoration tasks, such as deblurring and denoising, while further optimizing the architecture for mobile hardware deployment.
Summary updated