Abstract
Video Super-Resolution (VSR) aims to reconstruct high-resolution (HR) videos from low-resolution (LR) inputs while maintaining temporal consistency and visual realism. While diffusion models have set new benchmarks for generative quality, their iterative nature often leads to high computational costs and inter-frame flickering. This paper presents UltraVSR, a framework that achieves ultra-realistic VSR in a single diffusion step. By utilizing a Degradation-aware Reconstruction Scheduling (DRS) and a lightweight Recurrent Temporal Shift (RTS) module, UltraVSR significantly reduces inference time—achieving up to 35x speedup over existing diffusion-based methods—while maintaining state-of-the-art perceptual quality and temporal coherence.
1. Introduction
Video Super-Resolution (VSR) is a critical task in computer vision, focused on restoring high-fidelity details from degraded, low-resolution video sequences. Traditional deep learning approaches, while efficient, often struggle to generate the fine-grained textures required for "ultra-realistic" results, frequently producing over-smoothed or blurry outputs. Recently, Diffusion Models (DMs) have emerged as a powerful alternative, leveraging generative priors to synthesize realistic textures.
However, adapting DMs to the video domain introduces two significant hurdles: stochastic flickering and extreme computational overhead. The random nature of the denoising process often results in inconsistent textures across frames, while the iterative sampling required by standard diffusion processes makes high-resolution (2K/4K) video restoration impractical. UltraVSR addresses these challenges by streamlining the diffusion process into a single, efficient step without sacrificing the generative power of the underlying model.
2. Methodology
UltraVSR introduces four key innovations to enable efficient, high-quality video restoration.
2.1 Degradation-aware Reconstruction Scheduling (DRS)
Standard diffusion models typically start from pure Gaussian noise and perform dozens of iterative steps to reach a clean image. UltraVSR bypasses this by treating the VSR task as a direct mapping within the diffusion latent space. The DRS module estimates the degradation factor of the input LR video and maps it to a specific point on the diffusion noise schedule. This allows the model to perform a one-step reconstruction directly from the LR input to the HR output, drastically reducing the sampling time.
2.2 Recurrent Temporal Shift (RTS) Module
To ensure temporal consistency without the heavy computational cost of 3D convolutions or the unreliability of optical flow, UltraVSR utilizes the RTS module. This module is essentially parameter-free; it shifts a portion of the feature channels across adjacent frames. This simple mechanism allows the model to align and fuse temporal information effectively, ensuring that textures remain stable over time with minimal memory overhead.
2.3 Spatio-temporal Joint Distillation (SJD)
To further suppress flickering, the authors employ Spatio-temporal Joint Distillation during training. The model is guided by two Temporal Regularized UNets (TRUs)—one representing the distribution of real-world videos and the other representing the reconstructed sequences. By minimizing the discrepancy between their spatial and temporal gradients, the model learns to generate frames that are both spatially sharp and temporally fluid.
2.4 Temporally Asynchronous Inference (TAI)
To overcome the "memory wall" associated with long, high-resolution sequences, UltraVSR uses TAI. This strategy decouples spatial and temporal processing, allowing the model to handle sequences of up to 120 frames at 2K resolution on a single GPU. By processing spatial features in mini-batches before linking them temporally, the framework achieves an 8x throughput improvement over previous state-of-the-art models.
3. Experimental Results
UltraVSR was evaluated on standard benchmarks (REDS, SPMCS, UDM10) and the real-world VideoLQ dataset.
3.1 Efficiency and Speed
UltraVSR demonstrates remarkable efficiency compared to other diffusion-based VSR methods. On an NVIDIA RTX 8000, it processes a 720p frame in just 0.89 seconds, whereas competing models like STAR and MGLD-VSR require 14.5s and 31.8s, respectively. Furthermore, UltraVSR is highly lightweight, utilizing only 10.5M trainable parameters, a fraction of the 600M+ parameters found in some contemporary diffusion frameworks.
3.2 Visual Quality and Consistency
On the VideoLQ real-world benchmark, UltraVSR achieved the highest scores in non-reference metrics, including a +1.94 improvement in MUSIQ and a +0.014 increase in DOVER. These results indicate that the model not only generates superior realistic textures but also maintains higher temporal coherence than previous methods.
4. Conclusion and Future Work
UltraVSR represents a significant shift in video super-resolution, proving that the generative power of diffusion models can be harnessed in a single, efficient step. By replacing iterative sampling with degradation-aware scheduling and lightweight temporal shifting, the framework makes high-fidelity video restoration accessible for practical applications. Future research may focus on reducing the model's dependency on pretrained Stable Diffusion weights to mitigate inherent biases and extending the one-step paradigm to 8K resolutions.
Summary updated