Introduction
As display technology advances toward 4K and 8K Ultra-High Definition (UHD), the limitations of standard upscaling become apparent. Simply increasing spatial resolution (Super-Resolution or SR) often leaves viewers with "motion judder"—a choppy visual effect caused by low frame rates on large screens. To achieve a premium viewing experience, both spatial resolution and frame rate (Video Frame Interpolation or VFI) must be enhanced simultaneously.
Traditionally, these tasks are handled sequentially in a "cascaded" manner: either upscaling first then interpolating, or vice versa. However, these approaches are inherently flawed. Upscaling first increases pixel displacement, making motion tracking harder for VFI models. Conversely, interpolating first can introduce artifacts that are subsequently amplified by the SR model. This paper introduces FISRnet, a deep learning framework designed to solve both problems in a single, joint architecture.
Methodology: The Joint VFI-SR Framework
FISRnet is specifically engineered to convert legacy 2K 30 fps content into high-quality 4K 60 fps video. Unlike cascaded methods, it processes temporal and spatial information in parallel.
The architecture incorporates several key innovations:
- Joint Input/Output: The model accepts three consecutive Low-Resolution (LR) frames. It outputs a triplet of High-Resolution (HR) frames: a spatially upscaled version of the center frame and two synthesized "end-frames" that fill the temporal gaps.
- Multi-scale U-Net: To manage the large pixel displacements inherent in 4K video, the model employs a 3-level multi-scale structure, refining coarse predictions into high-fidelity outputs.
- Flow-Guided Synthesis: By utilizing bidirectional optical flow via a pre-trained PWC-Net, the network gains a deeper understanding of motion dynamics.
- Multi-scale Temporal Loss: This novel loss function is the framework's "secret sauce." By using a sliding window of five frames during training, the authors enforce temporal regularization, ensuring that overlapping predictions are consistent and motion remains fluid across time steps.
Key Results and Insights
The researchers evaluated FISRnet using a custom dataset of 4K 60 fps YouTube videos, focusing on scenes with high motion (displacements up to 124 pixels).
Quantitatively, FISRnet outperformed the best cascaded combinations (VFI → SR) by at least 1.42 dB in PSNR for interpolated frames. Qualitatively, the difference is striking. As shown in the visual comparisons, FISRnet successfully reconstructs sharp textures—such as the text on a moving boat, water ripples, or the intricate pattern on a basketball—that cascaded methods typically leave blurry or distorted.
A critical insight from this work is that performing VFI and SR together avoids the error accumulation found in multi-stage pipelines. Furthermore, the multi-scale temporal loss effectively eliminates the flickering common in per-frame optimization, resulting in temporally coherent video that appears natural to the human eye.
Conclusion and Future Work
While FISRnet demonstrates superior performance in joint VFI-SR tasks, it does face some trade-offs. Dedicated SR-only models can occasionally achieve higher PSNR on non-interpolated frames, and the computational demand of processing 4K 60 fps video remains high (averaging 2.73 seconds per sample on an NVIDIA TITAN Xp).
Future research may focus on optimizing the framework for real-time consumer applications. Additionally, the authors suggest that the multi-scale temporal loss is a generalizable tool that could improve other video restoration tasks, such as deblurring and denoising.
Summary updated