Multi-Scale Video Frame-Synthesis Network with Transitive Consistency Loss

Zhe Hu, Yinglan Ma, Lizhuang MaarXiv 2017 · arXiv

Introduction

Video frame synthesis, encompassing both interpolation (filling gaps between frames) and extrapolation (predicting future frames), is a fundamental challenge in computer vision. Traditional methods relying on explicit optical flow estimation are often computationally intensive and prone to visual artifacts when flow estimation fails. While recent deep learning models have moved toward flow-free architectures, they often remain rigid—hardcoded for specific time steps—and computationally heavy, limiting their utility on mobile or edge devices.

This paper introduces the Multi-Scale Frame-Synthesis Network (MSFSN), a unified generative adversarial network (GAN) designed to synthesize frames at any arbitrary time ratio using a single, compact model. By treating the target time as a continuous input variable and introducing a novel transitive consistency loss, the authors achieve state-of-the-art performance with significantly fewer parameters than existing solutions.

The Multi-Scale Architecture

The MSFSN frames synthesis as a continuous mapping problem. Unlike previous architectures that are fixed to a specific temporal midpoint (e.g., $t=0.5$), this network accepts a relative time ratio as an input channel. This allows the same model to perform interpolation ($0 < t < 1$) and extrapolation ($t > 1$) seamlessly.

The Multi-Scale Frame-Synthesis Network (MSFSN) architecture uses shared residual blocks across pyramid levels to maintain a compact parameter footprint.

To handle large motions efficiently, the network employs a coarse-to-fine pyramid structure. A key innovation here is parameter sharing across scales: the sub-networks at different pyramid levels share the exact same weights. This design choice leverages the fact that the synthesis task is scale-invariant, allowing the model to scale its receptive field by adding pyramid levels during inference without increasing the total parameter count (fixed at 7.4 million parameters).

Transitive Consistency Loss

A major contribution of this work is the Transitive Consistency Loss. This regularizer enforces a logical temporal symmetry: if the network uses Frame A and Frame B to generate an intermediate Frame C, it should be able to use the generated Frame C and Frame B to reconstruct Frame A. This "reversibility" constraint anchors the learning process, particularly when training on arbitrary time intervals which can otherwise be unstable. The training objective combines this novel loss with standard pixel-wise L1 loss, VGG-based perceptual loss, and adversarial loss.

Experimental Results

The model was evaluated on several benchmarks, including the high-speed GOPRO dataset, UCF-101, and KITTI. Quantitatively, the MSFSN achieves a PSNR of 35.8 on UCF-101, outperforming flow-based methods like FlowNet2 and matching the performance of much larger models like AdapSC (which is nearly three times the size).

Ablation study (a) showing the impact of transitive consistency loss on PSNR stability and (b) user study results comparing the proposed method against state-of-the-art baselines.

As shown in the ablation study, the transitive consistency loss is critical for achieving high PSNR and stable training. Furthermore, a subjective user study involving 42 participants revealed a strong preference for the MSFSN over competing methods. The model effectively avoids the "ringing" and warping artifacts common in optical flow approaches, maintaining structural integrity even in complex scenes like road markings in the KITTI dataset.

Conclusion and Future Work

The MSFSN provides a highly efficient and flexible framework for video frame synthesis. By treating time as a continuous parameter and utilizing shared-weight pyramids, the authors deliver a model that is both high-performing and suitable for deployment on resource-constrained devices. While the current local window filtering can occasionally result in slightly softer images compared to correspondence-based methods, future work aims to integrate explicit pixel-routing mechanisms to further enhance sharpness while maintaining the model's unified, lightweight nature.

Summary updated

Abstract

Traditional approaches to interpolate/extrapolate frames in a video sequence require accurate pixel correspondences between images, e.g., using optical flow. Their results stem on the accuracy of optical flow estimation, and could generate heavy artifacts when flow estimation failed. Recently methods using auto-encoder has shown impressive progress, however they are usually trained for specific interpolation/extrapolation settings and lack of flexibility and In order to reduce these limitations, we propose a unified network to parameterize the interest frame position and therefore infer interpolate/extrapolate frames within the same framework. To achieve this, we introduce a transitive consistency loss to better regularize the network. We adopt a multi-scale structure for the network so that the parameters can be shared across multi-layers. Our approach avoids expensive global optimization of optical flow methods, and is efficient and flexible for video interpolation/extrapolation applications. Experimental results have shown that our method performs favorably against state-of-the-art methods.
BibTeX
@misc{hu2017multi,
  author = {Zhe Hu and Yinglan Ma and Lizhuang Ma},
  title = {{Multi-Scale Video Frame-Synthesis Network with Transitive Consistency Loss}},
  year = {2017},
  eprint = {1712.02874},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1712.02874},
}
Figure