Abstract
This paper introduces Deformable Separable Convolution (DSepConv), a novel approach to video frame interpolation (VFI) that unifies the strengths of flow-based and kernel-based methods. By learning adaptive kernels, offsets, and masks, the model can sample relevant pixels from across a frame, allowing it to handle large motions with high precision and significantly lower computational overhead than previous fixed-kernel methods.
The Challenge of Video Frame Interpolation
Video Frame Interpolation (VFI) aims to synthesize a non-existent middle frame between two consecutive video frames. Traditionally, researchers have followed two distinct paths:
- Flow-based methods: These estimate optical flow to warp pixels. While intuitive, they often struggle with occlusions, lighting changes, and complex non-linear motions, leading to visual artifacts.
- Kernel-based methods: These treat interpolation as a local convolution problem, predicting a pixel's value based on its neighbors. To handle large motions, these models require massive kernels (e.g., $51 \times 51$). This is computationally expensive—processing thousands of pixels to produce one—and the model remains strictly limited by the predefined kernel size. If an object moves beyond the kernel's reach, the model cannot "see" it.
The authors aim to break this "local neighborhood" constraint, allowing the model to find relevant pixels even when they have moved significantly across the screen.
Proposed Method: DSepConv
The core innovation is DSepConv, which integrates three distinct mechanisms into a single convolution process:
- Deformable Offsets: Inspired by deformable convolution networks, the model learns "offsets" that allow the sampling grid to warp and stretch. Instead of looking at a rigid square, the model can "reach out" to distant, relevant pixels.
- Separable Kernels: To keep the parameter count manageable, the model approximates a 2D convolution kernel by predicting two 1D kernels (horizontal and vertical). This reduces complexity from $O(n^2)$ to $O(2n)$.
- Modulation Masks: The model predicts a weight (mask) for each sampled pixel, acting as a built-in occlusion handler to ignore irrelevant or occluded pixels.
The architecture utilizes a U-Net encoder-decoder to extract features, which are then processed by four specialized sub-modules to estimate the kernels, offsets, and masks for every output pixel.
Key Insights and Efficiency
A major contribution of this work is the mathematical demonstration that both traditional flow-based warping and standard kernel-based interpolation are special cases of DSepConv. By setting the kernel size to $1 \times 1$, DSepConv essentially becomes a flow-based method with an occlusion mask.
Furthermore, DSepConv achieves superior results using only a $5 \times 5$ kernel (25 pixels) compared to previous state-of-the-art models like SepConv, which used $51 \times 51$ kernels (2,601 pixels). Because the deformable offsets ensure those 25 pixels are the right ones, the model effectively performs "non-local" processing, which is crucial for high-speed action where objects jump across the frame.
Experimental Results
The authors evaluated DSepConv on industry-standard benchmarks including Vimeo90K, UCF101, and the Middlebury Benchmark.
- State-of-the-Art Performance: On the Middlebury Evaluation set, DSepConv ranked 3rd out of over 160 algorithms at the time of submission.
- Quantitative Gains: On the UCF101 dataset, DSepConv achieved a PSNR of 35.08 dB, outperforming the baseline SepConv (34.78 dB) and other complex models like DAIN.
- Visual Quality: Qualitative results demonstrate that DSepConv is significantly better at preserving sharp edges of fast-moving objects where other methods produce blur or "ghosting" artifacts.
Limitations and Future Work
While highly effective, the current model is designed for single-step interpolation ($t=0.5$) and does not natively support arbitrary-time interpolation without recursive application. Additionally, while more efficient in pixel sampling, the model still requires ~21.8 million parameters. Future work may involve slimming the encoder-decoder for mobile applications and extending the architecture to utilize longer frame sequences for improved temporal stability.
Summary updated