Efficient Meshflow and Optical Flow Estimation from Event Cameras

Xinglong Luo, Ao Luo, Zhengning Wang, Chunyu Lin, Bing Zeng, Shuaicheng LiuCVPR 2024 · DOI

Introduction

Traditional frame-based RGB cameras often fail in challenging environments characterized by high-speed motion or low-light conditions, resulting in motion blur or loss of texture. Event cameras, which asynchronously record per-pixel brightness changes, address these limitations by offering high dynamic range and microsecond-level temporal resolution. However, processing event data for motion estimation presents a computational challenge. While dense optical flow provides pixel-level motion, it is often computationally prohibitive for real-time applications. Conversely, global homography is efficient but lacks the flexibility to model complex, non-planar scenes.

This paper proposes Meshflow as a balanced alternative. Meshflow represents motion at the vertices of a regular grid, providing a lightweight yet flexible motion field. To advance this field, the authors introduce the first framework for event-based meshflow estimation, including a novel high-resolution dataset (HREM) and an efficient architecture named EEMFlow.

Methodology

The proposed framework centers on EEMFlow, a lightweight encoder-decoder network designed for rapid meshflow prediction.

EEMFlow and Dilated Feature Correlation

A core innovation in EEMFlow is the Dilated Feature Correlation (DFC) layer. Standard correlation layers, which compute feature similarities to estimate displacement, are computationally expensive when searching over large windows. DFC optimizes this by employing a non-uniform sampling strategy: it samples densely near the center of the search window and more sparsely at greater distances. This allows the model to capture large displacements without the quadratic increase in computational cost.

Overview of the EEMFlow architecture for meshflow and optical flow estimation.

EEMFlow+ and the CDC Module

For applications requiring dense optical flow, the authors extend the base model into EEMFlow+. This version incorporates a Confidence-induced Detail Completion (CDC) module. During the upsampling of coarse motion maps to full resolution, traditional methods often produce blurred object boundaries. The CDC module utilizes a self-attention mechanism to refine these edges, ensuring sharp transitions and better alignment with physical object boundaries.

The HREM Dataset

The lack of high-resolution event datasets with diverse motion patterns led to the creation of the High-Resolution Event Meshflow (HREM) dataset. Developed using Blender, HREM consists of 100 virtual scenes and 28,000 samples at a resolution of $1280 \times 720$. It provides ground-truth labels for both dense optical flow and sparse meshflow, serving as a robust benchmark for training and evaluation.

Experimental Results

The performance of EEMFlow was evaluated against several state-of-the-art (SOTA) optical flow methods on both synthetic (HREM) and real-world (DSEC) datasets.

Efficiency and Accuracy

EEMFlow demonstrates a significant leap in efficiency, achieving an inference speed of 142.9 FPS. As shown in the performance analysis, EEMFlow is approximately 39 times faster than top-tier models like ERAFT while maintaining superior accuracy (lowest End-Point Error) on the HREM dataset.

Performance comparison of EEMFlow against state-of-the-art methods in terms of error and inference time.

Real-World Generalization

On the DSEC online benchmark, the EEMFlow+ variant achieved state-of-the-art performance for dense flow estimation. It outperformed significantly larger models in accuracy while remaining the fastest in its category at 39.2 FPS. Qualitative results further indicate that event-based meshflow is robust in scenarios where RGB-based methods fail, such as high-speed motion in low-light environments.

Conclusion

The introduction of EEMFlow and the HREM dataset marks a significant step toward real-time, robust motion estimation using event cameras. By leveraging the sparsity of meshflow and the efficiency of dilated correlations, the proposed framework enables high-performance motion tracking suitable for resource-constrained platforms like drones and mobile devices. Future work will explore extending this framework to 3D scene flow and integrating depth information for enhanced perspective handling.

Summary updated

BibTeX
@inproceedings{luo2024efficient,
  author = {Xinglong Luo and Ao Luo and Zhengning Wang and Chunyu Lin and Bing Zeng and Shuaicheng Liu},
  title = {{Efficient Meshflow and Optical Flow Estimation from Event Cameras}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01816},
}
Figure