Stable Long-Term Recurrent Video Super-Resolution

Benjamin Naoto Chiche, Arnaud Woiselle, Joana Frontera-Pons, Jean-Luc StarckCVPR 2022 · DOI · arXiv

Abstract

Recurrent neural networks (RNNs) are widely used in video super-resolution (VSR) due to their computational efficiency and ability to leverage temporal information. However, these models often exhibit instability when processing long, quasi-static video sequences, leading to the accumulation of high-frequency artifacts. This paper introduces Middle Recurrent Video Super-Resolution (MRVSR), a framework designed to ensure long-term stability using Lipschitz stability theory. By isolating the recurrent loop and enforcing a 1-Lipschitz constraint through Stable Rank Normalization, MRVSR prevents divergence while maintaining the high-frequency sharpening capabilities required for quality upscaling. Experimental results on long-duration datasets demonstrate that MRVSR eliminates common "hatching" artifacts and maintains consistent performance over thousands of frames.

Introduction

Video Super-Resolution (VSR) aims to reconstruct high-resolution (HR) frames from a sequence of low-resolution (LR) inputs. Recurrent architectures have become the standard for this task because they offer a theoretically infinite temporal receptive field without the massive memory overhead associated with batch-processing multiple frames simultaneously. Despite their success on standard benchmarks, these models face a significant hurdle: long-term instability.

When processing long sequences with minimal motion—such as surveillance footage or static nature shots—recurrent models tend to "over-sharpen" the same pixels repeatedly. This feedback loop causes high-frequency errors to compound, eventually leading to visual divergence and a collapse in image quality. This paper addresses this "stability paradox" by proposing a mathematically grounded framework for stable recurrent VSR.

The Stability Paradox in VSR

The core issue identified by the researchers is that VSR is inherently an expansive operation. To sharpen an image, a network must boost high-frequency components, which mathematically contradicts the requirements for global stability (where the system should dampen noise).

Most current VSR models are trained on short clips of roughly 7 to 12 frames. In these short windows, or in videos with significant motion, the hidden state of the recurrent network is frequently "reset" by new information. However, in quasi-static scenes, the lack of motion allows errors to accumulate indefinitely. To solve this, the authors turn to Lipschitz stability theory, seeking a way to make the recurrent loop contractive (dampening) without losing the ability to generate sharp details.

The MRVSR Framework

The proposed Middle Recurrent Video Super-Resolution (MRVSR) framework splits the VSR task into three distinct components to balance stability and performance:

  1. Input Network ($\xi$): Responsible for extracting initial features from the current LR frames. This part remains unconstrained.
  2. Recurrent Network ($\phi^L$): The core memory unit. This component is strictly constrained to be 1-Lipschitz using Stable Rank Normalization (SRNL). This ensures that the feedback loop is contractive and cannot "explode" over time.
  3. Output Network ($\psi$): Performs the final upscaling and deconvolution. Like the input network, this remains unconstrained to allow for the necessary high-frequency sharpening.

By isolating the stability constraint to the recurrent "middle" of the network, MRVSR achieves a temporal receptive field of approximately 28 frames—four times longer than typical sliding-window approaches—while remaining stable over thousands of frames.

Experimental Evaluation

To test long-term stability, the authors introduced the Quasi-Static Video Set and an extreme stress test called Sequence 1-XL, which contains over 8,700 frames.

Key Results

Standard recurrent models like RLSP and RSDN showed a dramatic decline in Peak Signal-to-Noise Ratio (PSNR) after the first 100 frames of static video as artifacts began to dominate the image. In contrast, MRVSR maintained a steady PSNR throughout the entire duration of the sequences.

Visual comparison of VSR methods on a long sequence. Note the 'hatching' artifacts in the RLSP result (c) compared to the stable MRVSR result (d).

Visually, MRVSR eliminates the "hatching" and "ghosting" artifacts that plague unconstrained models. While there is a minor "stability tax"—a slight reduction in peak performance on short, high-motion clips compared to purely unconstrained state-of-the-art models—the gains in reliability for real-world, long-duration video are substantial.

Conclusion

MRVSR represents a significant step forward in making recurrent VSR models practical for real-world deployment. By applying Lipschitz stability theory to the recurrent loop, the authors have tamed the feedback loop, ensuring that "marathon" video sequences no longer lead to catastrophic divergence. Future work may involve adaptive constraints that adjust based on the level of motion detected in the scene, further optimizing the balance between stability and sharpening power.

Summary updated

BibTeX
@inproceedings{chiche2022stable,
  author = {Benjamin Naoto Chiche and Arnaud Woiselle and Joana Frontera-Pons and Jean-Luc Starck},
  title = {{Stable Long-Term Recurrent Video Super-Resolution}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00091},
}
Figure