Video Frame Interpolation via Direct Synthesis with the Event-based Reference

Yuhan Liu, Yongjian Deng, Hao Chen, Zhen YangCVPR 2024 · OpenReview

Abstract

Video Frame Interpolation (VFI) has witnessed a surge in popularity due to its abundant downstream applications. Event-based VFI (E-VFI) has recently propelled the advancement of VFI. Thanks to the high temporal resolution benefits, event cameras can bridge the informational void present between successive video frames. Most state-of-the-art E-VFI methodologies follow the conventional VFI paradigm, which pivots on motion estimation between consecutive frames to generate intermediate frames through a process of warping and refinement. However, this reliance engenders a heavy dependency on the quality and consistency of keyframes, rendering these methods susceptible to challenges in extreme real-world scenarios, such as missing moving objects and severe occlusion dilemmas. This study proposes a novel E-VFI framework that directly synthesizes intermediate frames leveraging event-based reference, obviating the necessity for explicit motion estimation and substantially enhancing the capacity to handle motion occlusion. Given the sparse and inherently noisy nature of event data, we prioritize the reliability of the event-based reference, leading to the development of an innovative event-aware reconstruction strategy for accurate reference generation. Besides, we implement a bi-directional event-guided alignment from keyframes to the reference using the introduced E-PCD module. Finally, a transformer-based decoder is adopted for prediction refinement. Comprehensive experimental evaluations on both synthetic and real-world datasets underscore the superiority of our approach and its potential to execute high-quality VFI tasks.

Introduction

Video Frame Interpolation (VFI) is the process of synthesizing intermediate frames between existing keyframes to enhance video smoothness or create slow-motion effects. Traditional state-of-the-art methods typically rely on motion estimation, calculating pixel displacement between keyframes to "warp" them into new positions. However, this approach faces significant challenges during complex, non-linear motions or severe occlusions—scenarios where the "informational void" between frames is too vast for simple interpolation.

Event cameras, which record asynchronous brightness changes at micro-second intervals, provide a continuous stream of data that can fill this void. While recent event-based VFI (E-VFI) methods have emerged, many still use event data merely to refine traditional motion warping. This paper introduces a paradigm shift: direct synthesis using an event-based structural reference, effectively bypassing the pitfalls of explicit motion estimation.

Methodology

The proposed framework shifts the role of the event camera from a secondary "helper" to a primary "structural guide" through a three-stage process.

1. Event-Aware Reconstruction Strategy (EARS)

Instead of calculating flow, the system first generates a "structural reference"—a grayscale-like representation of moving objects at the target timestamp. To address the sparse and noisy nature of raw event data, the authors developed an Event-Aware Reconstruction Strategy (EARS). This strategy utilizes custom erosion and dilation masking to ensure that only high-confidence structural information is used for reference generation.

Visualization of raw event data. The sparse and noisy nature of these signals requires robust reconstruction strategies like EARS to create a reliable structural reference.

2. Event-based Alignment (E-PCD)

The core of the direct synthesis approach is the Event-based Pyramid, Cascading, and Deformable convolution (E-PCD) module. This module aligns the features of the original keyframes to the structural reference created in the first stage. Rather than warping the entire image based on estimated flow, E-PCD uses the event-based reference as a template to accurately "place" textures from the keyframes, significantly improving performance in occluded regions.

3. Transformer-based Refinement

Finally, a transformer-based decoder is employed to refine the synthesized frame. This stage cleans up artifacts and ensures visual consistency, particularly in static regions where the event camera provides no signal.

Experimental Evaluation and Results

The model was rigorously tested on synthetic datasets (Vimeo90k, GoPro, SNU-FILM) and real-world benchmarks (HQF, DAVIS240, HS-ERGB). Quantitative evaluations show that the proposed method consistently outperforms existing state-of-the-art (SOTA) models like TimeLens and CBMNet. On the challenging SNU-FILM Extreme dataset, the model achieved a 31.46 dB PSNR, a substantial improvement over the 28.99 dB of its closest competitor.

Qualitative comparison of VFI methods. The proposed approach (e) demonstrates superior reconstruction of fast-moving objects (e.g., the sword fighter's arm) compared to RIFE (a), UPRNet (b), TimeLens (c), and CBMNet (d), closely matching the Ground Truth (f).

Qualitative results highlight the method's ability to maintain sharp edges and fine details on fast-moving objects where other methods produce "ghosting" or warped distortions. By inverting the paradigm—using events for structure and frames for texture—the model successfully reconstructs details that are often blurred out by traditional warping techniques.

Conclusion

This research demonstrates that direct synthesis via event-based references offers a robust solution to the occlusion and motion estimation challenges in VFI. While the method shows a clear advantage in dynamic scenes, future work will focus on optimizing computational complexity for real-time deployment and enhancing performance in static background regions.

Summary updated

Abstract

Video Frame Interpolation (VFI) has witnessed a surge in popularity due to its abundant downstream applications. Event-based VFI (E-VFI) has recently propelled the ad-vancement of VFI. Thanks to the high temporal resolution benefits, event cameras can bridge the informational void present between successive video frames. Most state-of-the-art E-VFI methodologies follow the conventional VFI paradigm, which pivots on motion estimation between consecutive frames to generate intermediate frames through a process of warping and refinement. However, this reliance engenders a heavy dependency on the quality and consis-tency of keyframes, rendering these methods susceptible to challenges in extreme real-world scenarios, such as missing moving objects and severe occlusion dilemmas. This study proposes a novel E-VFI framework that directly synthesize intermediate frames leveraging event-based reference, obviating the necessity for explicit motion estimation and substantially enhancing the capacity to handle motion occlusion. Given the sparse and inher-ently noisy nature of event data, we prioritize the relia-bility of the event-based reference, leading to the development of an innovative event-aware reconstruction strategy for accurate reference generation. Besides, we implement a bi-directional event-guided alignment from keyframes to the reference using the introduced E-PCD module. Finally, a transformer-based decoder is adopted for prediction re-finement. Comprehensive experimental evaluations on both synthetic and real-world datasets underscore the superiority of our approach and its potential to execute high-quality VFI tasks.
BibTeX
@inproceedings{liu2024video,
  author = {Yuhan Liu and Yongjian Deng and Hao Chen and Zhen Yang},
  title = {{Video Frame Interpolation via Direct Synthesis with the Event-based Reference}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=mnr19v5awY},
}
Figure