TL;DR
This paper introduces a novel deep learning network for video frame rate up-conversion (VFRU) that progressively refines intermediate frames using feature-based residue learning. A key innovation is a finetuning procedure for optical flow estimation that doesn't require ground truth optical flows, leading to improved performance and faster training.
What Problem Does This Paper Address?
Generating high-frame-rate videos from lower-frame-rate sources is crucial for enhancing user experience in multimedia applications. This process, known as video frame rate up-conversion or frame interpolation, has been a long-standing challenge in computer vision. Existing methods often rely on optical flow estimation, but these networks are typically trained on synthetic data and may not generalize well to real-world scenarios or the specific demands of frame interpolation. Furthermore, finetuning these optical flow networks for interpolation can be complex and prone to collapse.
Methods
The proposed method builds upon a flow-based paradigm, utilizing the PWC-Net architecture for optical flow estimation. The core innovation lies in a two-stage approach:
- Finetuned Optical Flow Estimation: The authors propose a finetuning procedure for the optical flow estimation module that leverages differentiable warping operations. Crucially, this finetuning does not require ground truth optical flows. Instead, it minimizes warping errors between the input frames and their warped counterparts at the target time instant. This process is shown to adapt the optical flow network to the specific needs of frame interpolation, leading to more accurate motion estimation.
- Feature-based Progressive Residue Refinement: After obtaining the finetuned optical flows, the network extracts multi-scale deep features from the input frames. These features, along with the input frames, are warped to the target time instant. A series of "ConvBlocks" then progressively refine the synthesized intermediate frame by learning and adding "residue" feature volumes at different levels of detail. This progressive refinement allows for the gradual incorporation of finer details and correction of artifacts. A shared decoder reconstructs the final synthesized frames.
Datasets
- Training: The Vimeo90K training set was used for both finetuning the PWC-Net and training the entire network. This dataset consists of 51,312 frame triplets with a resolution of 256x448, where the first and last frames are used as inputs and the intermediate frame as ground truth.
- Testing: Two test sets were used:
- The Vimeo90K testset, containing 3,782 frame triplets.
- The Adobe240fps dataset, from which 777 frame triplets were extracted.
Key Insights and Ideas
- Finetuning Optical Flow Without Ground Truth: The ability to finetune optical flow networks for interpolation without requiring ground truth optical flows is a significant contribution. This simplifies the training process and makes the method more applicable to real-world scenarios where such ground truth is often unavailable.
- Progressive Refinement for Detail: The progressive refinement of intermediate frames using feature-based residues allows for a more detailed and artifact-free synthesis. This approach effectively builds up the final frame by adding finer details layer by layer.
- Efficiency of Finetuning: The proposed finetuning procedure not only improves performance but also shortens the overall training time and prevents network collapse, making the training process more stable and efficient.
- Simplicity in Handling Occlusions: The method uses brightness errors to tackle pixel overlaps and occlusions, a simpler approach compared to methods that employ additional networks for importance map prediction.
Key Results
- Quantitative Performance: The proposed method outperforms state-of-the-art methods by approximately 0.2-0.3 dB in PSNR on the Vimeo90K testset. For example, on the Vimeo90K dataset, the method achieved a PSNR of 35.86 dB and an SSIM of 0.9689, surpassing other methods like SMSP (35.49 dB PSNR, 0.9671 SSIM).
- Qualitative Performance: Visual comparisons show that the proposed method generates more plausible results with fewer deformations and blurriness, particularly in areas with subtle details like hair or car plates.
- Arbitrary Frame Rate Up-conversion: The method can up-convert videos to any arbitrary frame rate, unlike some methods that are limited to specific interpolation factors.
Limitations and Future Work
- Linear Motion Assumption: The paper acknowledges that, like most frame synthesis methods, their approach is based on a linear motion assumption. Inferring velocity and acceleration from only two frames is inherently ill-posed, and more complex motion scenarios might benefit from multi-frame approaches.
- Future Work: The authors suggest that investigating multi-frame interpolation methods could further improve performance in scenarios with complex motion.
Summary updated