Introduction
Video frame synthesis, encompassing both interpolation (filling gaps between frames) and extrapolation (predicting future frames), is a fundamental challenge in computer vision. Traditional methods relying on explicit optical flow estimation are often computationally intensive and prone to visual artifacts when flow estimation fails. While recent deep learning models have moved toward flow-free architectures, they often remain rigid—hardcoded for specific time steps—and computationally heavy, limiting their utility on mobile or edge devices.
This paper introduces the Multi-Scale Frame-Synthesis Network (MSFSN), a unified generative adversarial network (GAN) designed to synthesize frames at any arbitrary time ratio using a single, compact model. By treating the target time as a continuous input variable and introducing a novel transitive consistency loss, the authors achieve state-of-the-art performance with significantly fewer parameters than existing solutions.
The Multi-Scale Architecture
The MSFSN frames synthesis as a continuous mapping problem. Unlike previous architectures that are fixed to a specific temporal midpoint (e.g., $t=0.5$), this network accepts a relative time ratio as an input channel. This allows the same model to perform interpolation ($0 < t < 1$) and extrapolation ($t > 1$) seamlessly.
To handle large motions efficiently, the network employs a coarse-to-fine pyramid structure. A key innovation here is parameter sharing across scales: the sub-networks at different pyramid levels share the exact same weights. This design choice leverages the fact that the synthesis task is scale-invariant, allowing the model to scale its receptive field by adding pyramid levels during inference without increasing the total parameter count (fixed at 7.4 million parameters).
Transitive Consistency Loss
A major contribution of this work is the Transitive Consistency Loss. This regularizer enforces a logical temporal symmetry: if the network uses Frame A and Frame B to generate an intermediate Frame C, it should be able to use the generated Frame C and Frame B to reconstruct Frame A. This "reversibility" constraint anchors the learning process, particularly when training on arbitrary time intervals which can otherwise be unstable. The training objective combines this novel loss with standard pixel-wise L1 loss, VGG-based perceptual loss, and adversarial loss.
Experimental Results
The model was evaluated on several benchmarks, including the high-speed GOPRO dataset, UCF-101, and KITTI. Quantitatively, the MSFSN achieves a PSNR of 35.8 on UCF-101, outperforming flow-based methods like FlowNet2 and matching the performance of much larger models like AdapSC (which is nearly three times the size).
As shown in the ablation study, the transitive consistency loss is critical for achieving high PSNR and stable training. Furthermore, a subjective user study involving 42 participants revealed a strong preference for the MSFSN over competing methods. The model effectively avoids the "ringing" and warping artifacts common in optical flow approaches, maintaining structural integrity even in complex scenes like road markings in the KITTI dataset.
Conclusion and Future Work
The MSFSN provides a highly efficient and flexible framework for video frame synthesis. By treating time as a continuous parameter and utilizing shared-weight pyramids, the authors deliver a model that is both high-performing and suitable for deployment on resource-constrained devices. While the current local window filtering can occasionally result in slightly softer images compared to correspondence-based methods, future work aims to integrate explicit pixel-routing mechanisms to further enhance sharpness while maintaining the model's unified, lightweight nature.
Summary updated