Multi-Frequency Representation Enhancement with Privilege Information for Video Super-Resolution

Fei Li, Linfeng Zhang, Zikun Liu, Juan Lei, Zhenbo Li

Abstract

Video Super-Resolution (VSR) is a critical task in computer vision that involves restoring high-resolution (HR) frames from low-resolution (LR) sequences. Current methods often struggle to balance the need for long-range spatial-temporal modeling with the strict computational constraints of edge devices. This paper introduces MFPI, a novel VSR framework that achieves state-of-the-art performance through two key innovations: a Multi-Frequency Representation Enhancement (MFE) module and a Privilege Training (PT) strategy. By operating in the frequency domain, MFPI captures global dependencies more efficiently than Transformer-based architectures. Furthermore, the PT strategy allows the model to leverage HR information during training to optimize internal representations without incurring any additional cost during inference. Experimental results across multiple benchmarks demonstrate that MFPI outperforms existing methods in both accuracy and efficiency.

Introduction

The primary challenge in Video Super-Resolution lies in effectively utilizing information across both spatial and temporal dimensions. Traditional Convolutional Neural Networks (CNNs) are often limited by small, local receptive fields, which prevent them from capturing long-range spatial patterns or complex motions across frames. While Vision Transformers (ViTs) have emerged as a solution for global modeling, their high computational complexity ($O(N^2)$) makes them impractical for real-time applications on mobile or edge hardware.

MFPI addresses this gap by asking: Can we provide CNNs with the global "vision" of a Transformer without the massive computational overhead? The authors propose shifting certain operations to the frequency domain and utilizing a "cheat sheet" approach during training to guide the network toward more effective feature extraction.

Methodology

Multi-Frequency Representation Enhancement (MFE)

To overcome the limitations of local receptive fields, the MFE module processes video data in the frequency domain rather than solely in the spatial domain. This module is composed of three distinct branches designed to capture different types of information:

  1. Spatial-Frequency Enhancement (SFE): This branch utilizes Fast Fourier Transforms (FFT) and learnable filters. In the frequency domain, a single point represents global information from the spatial domain. This allows the model to achieve a "global receptive field" with a complexity of only $O(N \log N)$, significantly lower than the self-attention mechanisms used in Transformers.
  2. Energy-Frequency Enhancement (EFE): Using Discrete Cosine Transforms (DCT) and a specialized energy function, this branch models relationships between feature channels. This helps the network prioritize salient information and suppress the noise that often accumulates in multi-frame video sequences.
  3. Auxiliary Branch: A standard pair of depthwise convolutions is included to maintain the local inductive bias that CNNs are naturally proficient at.
The MFE module utilizes frequency-domain operations like FFT and learnable filters to achieve a global receptive field efficiently.

Privilege Training (PT)

Inspired by the concept of "Learning Using Privilege Information" (LUPI), the authors introduce a novel training strategy. During the training phase, the model is allowed to "peek" at the ground-truth HR frames through a lightweight encoder. A mixing parameter ($\gamma$) controls the amount of HR information integrated into the LR input. As training progresses, $\gamma$ is gradually reduced to zero. This "training wheels" approach guides the model toward a better mathematical minima early in the process. Crucially, because $\gamma$ reaches zero by the end of training, the model requires zero extra information or parameters during actual inference.

Experimental Evaluation

Datasets

The performance of MFPI was rigorously evaluated on four standard VSR benchmarks:

  • REDS4: Known for high-quality video and complex motion patterns.
  • Vimeo-90K / Vimeo-90K-T: A large-scale dataset for general VSR tasks.
  • Vid4: A classic benchmark for video restoration.
  • UDM10: Specifically used to evaluate performance under different degradation types, including Bicubic and Blur Downsampling (BD).

Results and Discussion

MFPI consistently outperformed eighteen previous state-of-the-art methods across all tested benchmarks. On the UDM10 dataset (BD degradation), MFPI achieved a 0.36 dB PSNR improvement over BasicVSR++, the previous gold standard.

Beyond accuracy, MFPI demonstrates exceptional efficiency. With approximately 7.3M parameters and a runtime of 76ms for a standard 180x320 input, it is significantly lighter and faster than competing Transformer-based models. Visual analysis confirms that MFPI is particularly adept at recovering fine, repeating textures—such as text on signs or intricate patterns in clothing—that other models typically blur.

MFPI achieves the highest PSNR while maintaining a low runtime and parameter count compared to other state-of-the-art VSR methods.

Conclusion and Future Work

MFPI represents a significant advancement in efficient Video Super-Resolution by combining frequency-domain global modeling with a privileged training strategy. While the model shows great promise on synthetic benchmarks, future work will focus on testing MFPI against real-world degradations and complex unknown noise. Additionally, the authors plan to explore the application of the MFE and PT components to other video restoration tasks, such as deblurring and denoising, while further optimizing the architecture for mobile hardware deployment.

Summary updated

Abstract

CNN’s limited receptive field restricts its ability to capture long-range spatial-temporal dependencies, leading to unsatisfactory performance in video super-resolution(VSR). To tackle this challenge, this paper presents a novel multi-frequency representation enhancement module (MFE) that performs spatial-temporal information aggregation in the frequency domain. Specifically, MFE mainly includes a spatial-frequency representation enhancement branch which captures the long-range dependency in the spatial dimension, and an energy frequency representation enhancement branch to obtain the inter-channel feature relationship. Moreover, a novel model training method named privilege training is proposed to encode the privilege information from high-resolution videos to facilitate model training. With these two methods, we introduce a new VSR model named MFPI, which outperforms state-of-the-art methods by a large margin while maintaining good efficiency on various datasets, including REDS4, Vimeo, Vid4, and UDM10.
BibTeX
@inproceedings{li2023multi,
  author = {Fei Li and Linfeng Zhang and Zikun Liu and Juan Lei and Zhenbo Li},
  title = {{Multi-Frequency Representation Enhancement with Privilege Information for Video Super-Resolution}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01177},
}
Figure