Tangentially Elongated Gaussian Belief Propagation for Event-Based Incremental Optical Flow Estimation

Jun Nagata, Yusuke SekikawaCVPR 2023 · OpenReview

Abstract

Event-based cameras offer ultra-low latency and high dynamic range, yet extracting full 2D optical flow from their asynchronous, sparse data remains a significant challenge. This paper introduces Tangentially Elongated Gaussian Belief Propagation (TEGBP), a novel framework that bridges the gap between fast, local normal-flow estimators and accurate, global dense models. By representing motion uncertainty through Tangentially Elongated Gaussians (TEGs) and performing incremental inference on a factor graph, TEGBP achieves state-of-the-art accuracy while maintaining the computational efficiency required for real-time, event-driven applications.

Introduction

Traditional frame-based cameras are inherently limited by fixed exposure times and frame rates, which often lead to motion blur or data redundancy. In contrast, event-based sensors, such as Dynamic Vision Sensors (DVS), detect intensity changes asynchronously at each pixel, providing a stream of events with microsecond-level temporal resolution. While these sensors are ideal for high-speed motion analysis, they suffer from the fundamental "aperture problem": a single event only provides information about motion perpendicular to a local edge, known as normal flow.

Recovering the "full flow"—the true 2D velocity of a pixel—typically requires integrating information across multiple edges or orientations. Existing solutions often fall into two categories: fast incremental algorithms that struggle with accuracy due to the aperture problem, and high-accuracy deep learning models that are non-incremental and computationally expensive. The authors propose TEGBP to address this trade-off, providing a sparse, incremental, and accurate solution for full flow estimation.

The TEGBP Framework

The core innovation of this work is the Tangentially Elongated Gaussian (TEG) model. Instead of treating a normal flow measurement as a deterministic vector, the authors model it as a 2D Gaussian distribution in velocity space.

Overview of the TEGBP framework, illustrating the mapping from image space events to Tangentially Elongated Gaussians (TEG) and the resulting full flow in velocity space.

As shown in Figure 1, the TEG is designed with a very low variance in the direction perpendicular to the edge (where the measurement is certain) and a high variance along the edge (where the aperture problem introduces uncertainty). This probabilistic representation allows the system to maintain the full 2D uncertainty of each event.

To estimate the global flow, the image is represented as a sparse factor graph. Pixels act as variable nodes, and events introduce measurement factors. A smoothness prior is enforced via prior factors between neighboring nodes, assuming that local motion is spatially consistent. The system uses Belief Propagation (BP) to pass messages between nodes, incrementally updating the "belief" of the flow at each pixel as new events arrive.

Key Insights and Probabilistic Modeling

The effectiveness of TEGBP stems from its ability to handle the aperture problem through marginalization. When multiple TEGs from different edge orientations are combined with a smoothness prior, the joint distribution naturally converges toward the true full flow. This approach avoids the pitfalls of naive averaging used in previous incremental methods, which often leads to blurred or incorrect velocity estimates.

Furthermore, the algorithm leverages the inherent sparsity of event data. By only performing computations at the locations of active events, TEGBP preserves the low-latency benefits of the event sensor. A coarse-to-fine scheme is also employed to handle large-scale motions and accelerate the propagation of information across the factor graph.

Experimental Evaluation

The authors evaluated TEGBP against several benchmarks, including simulated datasets (ESIM-bricks) and real-world sequences (MVSEC, DSEC, and DVS-stripes). The results demonstrate a significant performance leap over the previous state-of-the-art incremental method, ARMS.

Comparison of velocity estimation histograms between the previous state-of-the-art (ARMS) and the proposed TEGBP method.

On the MVSEC dataset, TEGBP reduced the Average Endpoint Error (AEE) by approximately 30-40% compared to ARMS. For instance, in the indoor_flying1 sequence, the AEE dropped from 1.71 to 1.14. Additionally, the percentage of outlier pixels was halved in many scenarios. Qualitative tests on the DVS-stripes sequence showed that TEGBP could successfully distinguish the velocities of adjacent objects with different motions, a task where previous methods failed due to over-smoothing.

Conclusion and Future Work

TEGBP represents a major step forward in event-based optical flow, offering a robust and efficient way to recover full 2D motion. However, some limitations remain. The method's accuracy is partially dependent on the quality of the initial normal flow estimate; noisy textures can degrade performance. Future research will explore the use of lightweight neural networks to refine these initial measurements and the implementation of discontinuity-aware smoothness priors to better handle object boundaries. Finally, demonstrating the algorithm on specialized data-flow hardware will be crucial for proving its real-world utility in edge computing environments.

Summary updated

Abstract

Optical flow estimation is a fundamental functionality in computer vision. An event-based camera, which asynchronously detects sparse intensity changes, is an ideal device for realizing low-latency estimation of the optical flow owing to its low-latency sensing mechanism. An existing method using local plane fitting of events could utilize the sparsity to realize incremental updates for low-latency estimation; however, its output is merely a normal component of the full optical flow. An alternative approach using a frame-based deep neural network could estimate the full flow; however, its intensive non-incremental dense operation prohibits the low-latency estimation. We propose tangentially elongated Gaussian (TEG) belief propagation (BP) that realizes incremental full-flow estimation. We model the probability of full flow as the joint distribution of TEGs from the normal flow measurements, such that the marginal of this distribution with correct prior equals the full flow. We formulate the marginalization using a message-passing based on the BP to realize efficient incremental updates using sparse measurements. In addition to the theoretical justification, we evaluate the effectiveness of the TEGBP in real-world datasets; it outperforms SOTA incremental quasi-full flow method by a large margin. (The code is available at https://github.com/DensoITLab/tegbp/).
BibTeX
@inproceedings{nagata2023tangentially,
  author = {Jun Nagata and Yusuke Sekikawa},
  title = {{Tangentially Elongated Gaussian Belief Propagation for Event-Based Incremental Optical Flow Estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=auh3xeUYCb},
}
Figure