DFVSR: Directional Frequency Video Super-Resolution via Asymmetric and Enhancement Alignment Network

Shuting Dong, Feng Lu, Zhe Wu, Chun Yuan

Abstract

Video Super-Resolution (VSR) aims to reconstruct high-resolution (HR) frames from low-resolution (LR) sequences. While recent frequency-based methods have shown promise in recovering fine textures, they often ignore the directional nature of motion and suffer from spatial information loss. We propose the Directional Frequency Video Super-Resolution (DFVSR) network, which decomposes frames into directional frequency representations. By utilizing an asymmetric U-shaped alignment network and a specialized Directional Frequency-Enhanced Alignment (DFEA) module, DFVSR achieves state-of-the-art performance with superior parameter efficiency compared to leading models like BasicVSR++.

Introduction

The primary goal of VSR is the restoration of high-fidelity details from degraded video sequences. Despite significant progress, current architectures face three main bottlenecks: the treatment of frequency components as static appearance descriptors regardless of motion direction, the inability to distinguish between informative and noisy frequency regions during alignment, and the loss of critical spatial details during the encoding phase. DFVSR addresses these issues by explicitly modeling motion as a combination of directional frequencies and employing an architecture designed for robust spatial recovery.

Methods

The DFVSR framework is built upon several key innovations designed to optimize feature propagation and alignment.

Directional Frequency Representation (DFR)

Using the Discrete Wavelet Transform (DWT), DFVSR decomposes input frames into four distinct components: a low-frequency base ($l$) and three high-frequency components representing horizontal ($h$), vertical ($v$), and diagonal ($d$) directions. This decomposition allows the network to process motion as an amalgamation of directional frequency changes rather than simple pixel displacement.

Asymmetrical U-shaped Unit (DFUnit)

The core processing block of the network is the DFUnit. As shown in Figure 1, the architecture follows an asymmetrical U-shaped design. The encoder progressively integrates high-frequency alignment features into the low-frequency base. Conversely, the decoder features an iterative refinement branch that maintains communication with the encoder to recover spatial information typically lost during downsampling.

Overview of the DFVSR architecture and the internal structure of the DFUnit.

Directional Frequency-Enhanced Alignment (DFEA)

To handle temporal alignment, the authors introduce DFEA. Unlike standard Deformable Convolution (DCN), DFEA employs "double enhancements" to prioritize high-fidelity regions. It leverages learned offsets from one directional component to guide others, ensuring more stable and robust alignment across the sequence.

Experiments and Results

DFVSR was evaluated on standard benchmarks including REDS, Vimeo-90K, Vid4, and UDM10, considering both Bicubic (BI) and Blur Downsampling (BD) degradations.

Quantitative Performance and Efficiency

DFVSR demonstrates remarkable efficiency, utilizing only 7.1M parameters—fewer than BasicVSR++ (7.3M) and significantly less than the transformer-based VSR-T (32.6M). Despite its smaller size, it achieved a PSNR of 32.76 dB on the REDS4 (BI) dataset, outperforming BasicVSR++ (32.39 dB).

Qualitative Analysis

Visual comparisons highlight DFVSR's ability to restore sharp edges and fine structural details, such as window slats and text, which are often blurred or distorted by competing models.

Qualitative comparison of DFVSR against other state-of-the-art VSR methods across various datasets.

Temporal Consistency

Beyond per-frame accuracy, temporal profile evaluations indicate that DFVSR maintains smoother and more consistent transitions over time compared to models like EDVR or BasicVSR, reducing flickering artifacts in reconstructed videos.

Conclusion

DFVSR introduces a paradigm shift by treating motion through the lens of directional frequency. By combining DWT-based decomposition with an asymmetric alignment network, the model achieves a new state-of-the-art in VSR performance while remaining highly parameter-efficient. Future work may involve optimizing inference latency and extending this directional frequency approach to related tasks like video deblurring and frame interpolation.

Summary updated

Abstract

Recently, techniques utilizing frequency-based methods have gained significant attention, as they exhibit exceptional restoration capabilities for detail and structure in video super-resolution tasks. However, most of these frequency-based methods mainly have three major limitations: 1) insufficient exploration of object motion information, 2) inadequate enhancement for high-fidelity regions, and 3) loss of spatial information during convolution. In this paper, we propose a novel network, Directional Frequency Video Super-Resolution (DFVSR), to address these limitations. Specifically, we reconsider object motion from a new perspective and propose Directional Frequency Representation (DFR), which not only borrows the property of frequency representation of detail and structure information but also contains the direction information of the object motion that is extremely significant in videos. Based on this representation, we propose a Directional Frequency-Enhanced Alignment (DFEA) to use double enhancements of task-related information for ensuring the retention of high-fidelity frequency regions to generate the high-quality alignment feature. Furthermore, we design a novel Asymmetrical U-shaped network architecture to progressively fuse these alignment features and output the final output. This architecture enables the intercommunication of the same level of resolution in the encoder and decoder to achieve the supplement of spatial information. Powered by the above designs, our method achieves superior performance over state-of-the-art models on both quantitative and qualitative evaluations.
BibTeX
@inproceedings{dong2023dfvsr,
  author = {Shuting Dong and Feng Lu and Zhe Wu and Chun Yuan},
  title = {{DFVSR: Directional Frequency Video Super-Resolution via Asymmetric and Enhancement Alignment Network}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/76},
}
Figure