Introduction
Video Frame Interpolation (VFI) is a critical task in computer vision aimed at synthesizing intermediate frames to enhance video smoothness and frame rates. While Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have traditionally dominated this field, they face significant trade-offs between receptive field size and computational complexity.
Recently, the Mamba architecture, based on State Space Models (SSMs), has emerged as a promising linear-time alternative to Transformers. However, adapting Mamba for 2D and 3D visual data introduces challenges such as spatial discontinuity—where flattening a 2D image into a 1D sequence breaks local pixel relationships—and historical decay, where the influence of early tokens diminishes over long sequences. LC-Mamba addresses these issues through a novel scanning strategy and window-based processing.
Methodology
Overall Architecture
LC-Mamba follows a comprehensive pipeline consisting of a Low-Level Feature Extractor (LFE), a Spatiotemporal Feature Extractor (STFE), a Motion Estimator (ME), and a Residual Refiner (RR). The core innovation lies within the STFE, which utilizes LC-Mamba blocks to capture fine-grained motion and long-range dependencies efficiently.
Hilbert Curve-based Selective Scan (H-SS2D)
To preserve spatial correlations during the 1D scanning process, LC-Mamba employs a Hilbert curve path. Unlike standard raster scanning, the Hilbert curve ensures that pixels adjacent in 2D space remain close in the 1D sequence. This "continuous" scanning pattern significantly improves the model's ability to capture local textures and motion boundaries.
Shifted Window Technique
To mitigate the effects of historical decay in long sequences, LC-Mamba restricts scanning to local windows. To enable information exchange between these windows, the model adopts a Shifted Window approach similar to the Swin Transformer. By shifting the window partition in successive layers, the model effectively expands its receptive field and eliminates boundary artifacts.
Spatiotemporal Modeling (H-SS3D)
For VFI, capturing motion between frames is essential. LC-Mamba introduces Voxel-level 3D Scanning (H-SS3D), which interleaves tokens from two consecutive frames along the Hilbert path. This allows the SSM to model spatiotemporal relationships simultaneously, leading to more accurate motion estimation.
Experiments and Results
Performance Benchmarks
The model was rigorously evaluated on datasets including Vimeo90K, UCF101, Middlebury, and SNU-FILM. LC-Mamba achieved state-of-the-art results, notably reaching a PSNR of 36.53 dB on Vimeo90K, outperforming previous Mamba-based models like VFIMamba.
Efficiency and Scalability
A key strength of LC-Mamba is its parameter efficiency. The "Efficient" variant (Ours-E) utilizes only 6.7M parameters, yet it outperforms significantly larger Transformer-based models. Furthermore, the window-based Hilbert scan demonstrates excellent scalability, maintaining high performance on 4K resolutions (Xiph dataset) where global attention mechanisms typically become computationally prohibitive.
Conclusion
LC-Mamba successfully adapts the Mamba architecture for high-performance video frame interpolation. By combining Hilbert curve scanning with shifted local windows, it overcomes the inherent limitations of 1D SSMs for 2D/3D data. The resulting model is both highly accurate and remarkably efficient, setting a new benchmark for resource-constrained, high-resolution video processing. Future work will explore deeper integration of global context to further enhance long-range motion modeling.