VSRM: A Robust Mamba-Based Framework for Video Super-Resolution

Dinh Phu Tran, Dao Duy Hung, Daeyoung Kim

TL;DR

This paper introduces VSRM, a novel video super-resolution (VSR) framework that leverages the Mamba architecture to overcome the limitations of existing CNN and Transformer-based methods. VSRM achieves state-of-the-art results by introducing specialized Mamba blocks for spatio-temporal feature extraction, a deformable alignment module for handling motion, and a frequency-domain loss for better high-frequency detail recovery.

What Problem Does This Paper Address?

Video Super-Resolution (VSR) aims to enhance the resolution of low-resolution videos. This task is challenging because it requires processing long sequences of frames to effectively capture inter-frame information and maintain temporal consistency. Existing methods face trade-offs: CNNs are limited by their local receptive fields, struggling to capture long-range dependencies, while Transformers, despite their power, suffer from quadratic complexity, making them computationally expensive for long sequences. Recently, Mamba has emerged as a promising alternative, offering linear complexity and long-range modeling capabilities, but its application to VSR was unexplored. This paper addresses the need for a VSR framework that can efficiently handle long sequences, extract rich spatio-temporal features, and accurately align frames with complex motion, all while maintaining high visual quality.

Methods

The VSRM framework is built upon several novel components:

  • Dual Aggregation Mamba Block (DAMB): This is the core of VSRM, comprising:
  • Spatial-to-Temporal Mamba Blocks (S2TMBs): These blocks process sequences by scanning first spatially and then temporally, enabling both spatial and temporal awareness. They use bidirectional scanning (forward and backward) to capture comprehensive spatio-temporal information.
  • Temporal-to-Spatial Mamba Block (T2SMB): This block prioritizes temporal scanning over spatial scanning, further enhancing the extraction of temporal information. The authors found that using only a forward scan in T2SMB yields better results and efficiency.
  • Temporal Gated Feed-forward Network (TGFN): This module enhances the standard feed-forward network by incorporating 3D depthwise convolutions and a gating mechanism. This allows for more effective modeling of spatio-temporal relationships and better feature representation.
  • Deformable Cross-Mamba Alignment (DCA): This module addresses the challenge of aligning adjacent frames, which can be difficult with significant object motion. DCA uses a pre-trained SpyNet for initial flow estimation and then employs a deformable window scheme with a cross-mamba module. This allows for more flexible and adaptive warping operations, enabling better alignment even in cases of minimal or excessive motion.
  • Frequency Charbonnier-like Loss (FCL): To improve the recovery of high-frequency details (like edges and textures), VSRM introduces FCL. Unlike traditional pixel-wise losses that can lead to overly smooth results, FCL operates in the frequency domain by calculating the loss on the Real and Imaginary parts of the Fourier transform. This approach ensures that both low- and high-frequency components are balanced, leading to sharper and more visually appealing results.

Datasets

The paper uses the following datasets:

  • Training: REDS [40] and Vimeo-90K [55].
  • Testing: REDS4 (a subset of REDS), Vimeo-90K-T, and Vid4 [35].

The authors note that REDS and Vimeo-90K have different motion characteristics, with REDS featuring larger motions and Vimeo-90K having smaller movements, allowing for comprehensive evaluation.

Key Insights and Ideas

  • Mamba's Suitability for VSR: The paper convincingly demonstrates that Mamba's linear complexity and long-range modeling capabilities are a significant advantage for VSR, overcoming key limitations of both CNNs and Transformers.
  • Synergy of Spatial and Temporal Mamba: The combination of S2TMBs and T2SMBs within the DAMB is crucial for extracting rich spatio-temporal features, highlighting the importance of a structured approach to processing video data.
  • Deformable Alignment for Motion Robustness: The DCA module's ability to adapt to varying pixel movements through a deformable window and cross-mamba mechanism is a key innovation for handling real-world video scenarios.
  • Frequency Domain Loss for Detail Recovery: FCL's effectiveness in recovering high-frequency details by operating on the Real and Imaginary parts of the Fourier transform is a significant contribution to improving visual quality in VSR.

Key Results

  • State-of-the-Art Performance: VSRM achieves state-of-the-art results on multiple benchmarks (REDS4, Vimeo-90K-T, Vid4) for 4x upscaling, outperforming existing methods like VRT, PSRT-recurrent, and IART.
  • Improved PSNR/SSIM: VSRM shows significant improvements in PSNR and SSIM, with gains of up to 0.83dB and 0.90dB over VRT on REDS4.
  • Efficiency: Despite its advanced capabilities, VSRM maintains a competitive parameter count and computational cost, balancing performance with efficiency.
  • Qualitative Improvements: Visual comparisons show VSRM producing clearer and more precise results, accurately restoring fine details like license plate numbers and textures, which are often lost in other methods.

Limitations and Future Work

  • Mamba Optimization: The authors acknowledge that the acceleration and optimization of Mamba are still active research areas, suggesting potential for further improvements in VSRM's efficiency.
  • Adaptability to Other Tasks: VSRM's success suggests its potential adaptability to other low-level vision tasks such as video colorization, deblurring, and denoising, which could be explored in future work.
  • Pre-trained SpyNet Dependency: The DCA module relies on a pre-trained SpyNet for flow estimation. While effective, exploring end-to-end trainable alignment or alternative flow estimation methods could be a future direction.

Summary updated

Abstract

Video super-resolution remains a major challenge in low-level vision tasks. To date, CNN- and Transformer-based methods have delivered impressive results. However, CNNs are limited by local receptive fields, while Transformers struggle with quadratic complexity, posing challenges for processing long sequences in VSR. Recently, Mamba has drawn attention for its long-sequence modeling, linear complexity, and large receptive fields. In this work, we propose VSRM, a novel \textbf{V}ideo \textbf{S}uper-\textbf{R}esolution framework that leverages the power of \textbf{M}amba. VSRM introduces Spatial-to-Temporal Mamba and Temporal-to-Spatial Mamba blocks to extract long-range spatio-temporal features and enhance receptive fields efficiently. To better align adjacent frames, we propose Deformable Cross-Mamba Alignment module. This module utilizes a deformable cross-mamba mechanism to make the compensation stage more dynamic and flexible, preventing feature distortions. Finally, we minimize the frequency domain gaps between reconstructed and ground-truth frames by proposing a simple yet effective Frequency Charbonnier-like loss that better preserves high-frequency content and enhances visual quality. Through extensive experiments, VSRM achieves state-of-the-art results on diverse benchmarks, establishing itself as a solid foundation for future research.
BibTeX
@inproceedings{tran2025vsrm,
  author = {Dinh Phu Tran and Dao Duy Hung and Daeyoung Kim},
  title = {{VSRM: A Robust Mamba-Based Framework for Video Super-Resolution}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure