Abstract
High-resolution video frame interpolation (VFI) is a computationally demanding task, often restricted by the "memory wall" of modern hardware. This paper presents fLDR-Net, a lightweight architecture designed specifically for 4K content. By utilizing a trainable, PCA-inspired dimensionality reduction layer (fLDR), the model compresses image representations early in the pipeline. This approach reduces memory usage by over 60% and parameter counts by 80% compared to existing state-of-the-art methods, while maintaining high-quality synthesis on challenging 4K benchmarks.
Introduction
Video Frame Interpolation (VFI) aims to synthesize intermediate frames between existing ones, enabling smoother motion and high-quality slow-motion effects. While deep learning has significantly advanced VFI for low-resolution video, 4K content presents a unique challenge. Standard Convolutional Neural Networks (CNNs) typically extract features from full-resolution images, which for 4K frames requires massive VRAM and computational power.
Existing solutions often rely on downscaling—which sacrifices detail—or massive synthesis networks that are impractical for consumer-grade GPUs. fLDR-Net addresses this by introducing an early compression mechanism that preserves high-frequency details necessary for motion estimation while drastically lowering the memory footprint.
Methodology
Finetuned Linear Dimensionality Reduction (fLDR)
The core of the proposed architecture is the fLDR layer. Traditional VFI models use deep CNN layers for feature extraction. In contrast, fLDR-Net utilizes block-based Principal Component Analysis (PCA) but with a critical modification: the PCA projection vectors (eigenvectors) and mean vectors are treated as trainable parameters.
By making these basis vectors learnable, the network can discover a projection space optimized specifically for interpolation rather than general variance. This allows the model to prioritize features like edges and contrast, which are vital for accurate motion estimation, even within a compressed representation.
Early Compression and Synthesis-Free Design
The architecture processes images by dividing them into $8 \times 8$ blocks, which are immediately projected into a lower-dimensional space. This early compression ensures that subsequent modules—specifically flow estimation and occlusion estimation—operate on a reduced spatial footprint.
To further enhance efficiency, fLDR-Net eliminates the heavy "synthesis network" common in other VFI models. Instead, it uses a lightweight learned weighting map. The model warps the original frames based on estimated motion and fuses them using a softmax-based occlusion mask. This design choice, combined with parameter sharing across the image pyramid, results in a model with only 0.9 million parameters.
Experimental Setup and Datasets
The model was trained on X-Train, a dataset consisting of 4K video crops. Evaluation was conducted across several benchmarks to test different motion profiles:
- Xiph-4K: A standard benchmark characterized by relatively low motion.
- X-Test: A benchmark featuring high motion but limited scene variety.
- Inter4K-S & Inter4K-L: Two new benchmarks introduced by the authors. Inter4K-S focuses on standard motion, while Inter4K-L contains statistically larger motions, providing a rigorous test for motion estimation accuracy in 4K scenes.
Results and Discussion
Memory and Parameter Efficiency
fLDR-Net demonstrates significant improvements in hardware accessibility. It requires only 4.6 GB of VRAM to process a 4K frame, whereas competing models like XVFI require upwards of 12 GB. This makes fLDR-Net viable for a much wider range of consumer-grade hardware. Furthermore, its parameter count (0.9M) is a fraction of that found in models like RIFE (5.5M) or XVFI (18M).
Accuracy and Performance
On the challenging Inter4K-L benchmark, fLDR-Net achieved a PSNR of 25.16, outperforming other methods that do not utilize heavy, pretrained optical flow networks. The model maintains competitive inference speeds of approximately 0.5 seconds per 4K frame, largely due to the removal of the heavy convolutional overhead in the synthesis stage.
Limitations
Despite its efficiency at high resolutions, the $8 \times 8$ block compression is too aggressive for low-resolution datasets like Vimeo90K, where fLDR-Net performs significantly worse than traditional CNNs. Additionally, the current inference time is bottlenecked by the non-learnable warping module. Finally, the block-based approach can occasionally introduce subtle artifacts at block boundaries, which remains an area for future refinement.
Conclusion
fLDR-Net proves that high-resolution VFI does not strictly require massive synthesis networks or high-end server GPUs. By integrating trainable dimensionality reduction directly into the feature extraction process, the model achieves a balance of efficiency and quality that paves the way for practical 4K video processing on consumer devices.
Summary updated