EPA: Boosting Event-based Video Frame Interpolation with Perceptually Aligned Learning

Yuhan Liu, LingHui Fu, Zhen Yang, Hao Chen, Youfu Li, Yongjian DengNeurIPS 2025 · OpenReview

TL;DR

EPA (Perceptually Aligned Learning) is a new framework for event-based video frame interpolation that ditches traditional pixel-level supervision in favor of feature-level learning. By aligning semantic-perceptual features rather than raw pixels, the model remains robust even when input frames are blurry or noisy, producing intermediate frames that are significantly more consistent with human visual preferences.

What Problem Does This Paper Address?

Event-based Video Frame Interpolation (E-VFI) uses the high temporal resolution of event cameras—sensors that only record changes in brightness—to fill in the gaps between standard video frames. This is crucial for capturing high-speed motion or non-rigid deformations where standard cameras fail.

However, there is a "dirty secret" in the field: the very scenarios where E-VFI is most needed (high-speed motion) are exactly where standard "keyframe" images suffer from severe motion blur and sensor noise. Most existing models rely on direct pixel-level supervision, essentially trying to force the model to match "ground truth" images that might themselves be degraded. This leads to poor generalization and "hallucinated" artifacts.

Comparison of previous image-level supervision vs. EPA's feature-level paradigm

The authors argue that while pixels might be messy, the semantic meaning of a scene remains relatively stable even under degradation. EPA shifts the paradigm from image-level supervision to feature-level supervision to bypass these pixel-level degradations.

Methods

The authors introduce a two-stage framework designed to decouple the learning process from unreliable pixel data:

EPA's two-stage training framework
  1. Stage 1: Perceptual Feature Extraction & Reconstruction:
    • They use a Vision Foundation Model (VFM)—specifically a DINO-pretrained ResNet—to extract high-level semantic features.
  2. Because VFMs often ignore low-level details (like the texture of the sky), they introduce a Style Adapter. This lightweight module supplements the semantic features with low-level cues to ensure the final image isn't just semantically correct, but also visually rich.
  3. A Reconstruction Generator is trained to turn these augmented features back into high-fidelity images.
  4. Stage 2: Bidirectional Event-Guided Alignment (BEGA):
    • To create an intermediate frame at a specific time, the model must align features from the surrounding keyframes.
  5. The BEGA module uses event voxel grids (processed event data) to guide deformable convolutions. This allows the model to "warp" the semantic features of the keyframes to the target timestamp using the precise motion timing provided by the event camera.
  6. Crucially, the supervision happens in the feature space. The model learns to predict the features of the ground truth, rather than the pixels, making it "degradation-insensitive."

Datasets

The researchers evaluated EPA on a mix of synthetic and real-world benchmarks:

  • Synthetic: Vimeo90k-Triplet and GOPRO (where events are simulated).
  • Real-World:
    • HS-ERGB: High-speed sequences with various motion types.
  • BS-ERGB: Characterized by significant noise and complex non-rigid deformations.
  • EventAid-F: A dataset containing diverse real-world motion scenarios like buildings and sculptures.

Key Insights and Ideas

  • The Feature Space Advantage: The core "aha moment" is that the L2 distance between a blurry image and a sharp image is huge in pixel space, but relatively small in the feature space of a Vision Foundation Model. Training in feature space acts as a natural "denoiser" and "deblurrer."
  • DINO isn't enough: While DINO is great at identifying what is in a scene, it's bad at style (color, texture, non-salient regions). The Style Adapter is the bridge that allows a high-level semantic model to perform a low-level synthesis task.
  • Decoupling is Key: By freezing the reconstruction generator and focusing on feature alignment in the second stage, the model avoids the "overfitting" to low-quality inputs that plagues end-to-end pixel-based models.

Key Results

EPA consistently outperformed state-of-the-art methods (like TimeLens and CBMNet) across almost all perceptual metrics:

  • Perceptual Quality: On the GOPRO dataset (15-skip), EPA achieved an LPIPS score of 0.008, significantly better than the next best competitor, Timelens (0.012).
  • Visual Realism: Qualitative results show that EPA handles "extreme" cases—like fast-moving fingers or splashing water—much better than previous methods, which often produced "ghosting" or severe blur in those areas.
  • Generalization: Because it relies on semantic features, EPA showed much stronger performance when moving from synthetic training data to messy, real-world sensor data.

Limitations and Future Work

While the paper demonstrates a major leap in perceptual quality, some limitations remain:

  • Two-Stage Complexity: The training process is multi-stage and relies on a large, pre-trained VFM, which might be computationally heavy compared to simpler end-to-end architectures.
  • Stochasticity: Like many generative-adjacent methods, there is a trade-off between pixel-perfect accuracy (PSNR) and perceptual realism (LPIPS). In some extreme cases, the model might prioritize a "realistic-looking" texture over an exact pixel match to the ground truth.
  • Future Directions: The authors suggest that integrating more advanced VFMs or exploring more efficient "style" transfer mechanisms could further refine the balance between semantic consistency and fine-grained detail.

Summary updated

Abstract

Event cameras, with their capacity to provide high temporal resolution information between frames, are increasingly utilized for video frame interpolation (VFI) in challenging scenarios characterized by high-speed motion and significant occlusion. However, prevalent issues of blur and distortion within the keyframes and ground truth data used for training and inference in these demanding conditions are frequently overlooked. This oversight impedes the perceptual realism and multi-scene generalization capabilities of existing event-based VFI (E-VFI) methods when generating interpolated frames. Motivated by the observation that semantic-perceptual discrepancies between degraded and pristine images are considerably smaller than their image-level differences, we introduce EPA. This novel E-VFI framework diverges from approaches reliant on direct image-level supervision by constructing multilevel, degradation-insensitive semantic perceptual supervisory signals to enhance the perceptual realism and multi-scene generalization of the model's predictions. Specifically, EPA operates in two phases: it first employs a DINO-based perceptual extractor, a customized style adapter, and a reconstruction generator to derive multi-layered, degradation-insensitive semantic-perceptual features ($\mathcal{S}$). Second, a novel Bidirectional Event-Guided Alignment (BEGA) module utilizes deformable convolutions to align perceptual features from keyframes to ground truth with inter-frame temporal guidance extracted from event signals. By decoupling the learning process from direct image-level supervision, EPA enhances model robustness against degraded keyframes and unreliable ground truth information. Extensive experiments demonstrate that this approach yields interpolated frames more consistent with human perceptual preferences. *The code will be released upon acceptance.*
BibTeX
@inproceedings{liu2025epa,
  author = {Yuhan Liu and LingHui Fu and Zhen Yang and Hao Chen and Youfu Li and Yongjian Deng},
  title = {{EPA: Boosting Event-based Video Frame Interpolation with Perceptually Aligned Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=zxZPpVoCNO},
}
Figure