Optical Flow Estimation by Matching Time Surface with Event-Based Cameras

Jun Nagata, Yusuke Sekikawa, Yoshimitsu AokiSensors 2021 · OpenReview

TL;DR

This paper introduces a novel method for estimating dense optical flow from event-based cameras by matching "time surfaces." Their new "Surface Matching Loss" function offers a more stable loss landscape and more accurate gradients compared to existing methods, leading to improved optical flow estimation without needing to reconstruct image intensity.

What Problem Does This Paper Address?

Estimating optical flow, which describes the motion of pixels between consecutive frames, is a fundamental task in computer vision. Traditional cameras capture images at fixed frame rates, which can lead to motion blur and missed fast events. Event-based cameras, inspired by biological vision, overcome these limitations by asynchronously reporting pixel-level brightness changes with precise timestamps. This high temporal resolution and dynamic range make them ideal for capturing fast motion. However, the unique, sparse, and asynchronous output of event cameras presents a significant challenge for existing optical flow algorithms, which are designed for dense, frame-based data. Specifically, previous methods often rely on reconstructing image intensity or making assumptions about brightness constancy, which can be problematic with event data. This paper aims to estimate dense optical flow directly from event data without these limitations.

Methods

The core of this work is a new loss function called "Surface Matching Loss." This method leverages the "time surface," a representation where each pixel stores the timestamp of its most recent event. The key idea is to measure the consistency of timestamps between the current time surface and a time surface that has been shifted by a small time interval ($\Delta t$).

Here's a breakdown:

  • Time Surface: For each pixel and event polarity (positive/negative brightness change), the time surface $S_p(x)$ stores the timestamp of the latest event. This effectively summarizes the event stream at each pixel.
  • Shifted Time Surface: A shifted time surface $S'_p(x)$ is created by subtracting $\Delta t$ from the original time surface: $S'_p(x) = S_p(x) - \Delta t$.
  • Surface Matching Loss: The proposed loss function, $L_{surface}(v)$, is defined by minimizing the difference between the current time surface $S_p(x)$ and the shifted time surface warped by the estimated optical flow $v$: $S_p(x) \approx S'_p(x + v(x))$. This essentially checks if events are temporally consistent across a small motion.
  • Smoothness Regularization: To ensure smooth optical flow fields, an L1 smoothness regularization term, $L_{smoothness}(v)$, is added. This term penalizes large differences in optical flow between neighboring pixels, while still allowing for discontinuities at object edges.
  • Optimization: The final cost function is a combination of the Surface Matching Loss and the smoothness regularization, which is then minimized to estimate the optical flow $v$.

This approach differs from prior work by directly using the temporal information encoded in the time surface, avoiding the need to reconstruct image intensity or rely on brightness constancy assumptions.

Datasets

The paper evaluates their method on the following datasets:

  • ESIM (Event-based Simulator): Used for generating synthetic event data and ground truth optical flow for testing the loss landscape and initial experiments.
  • MVSEC (Multi-Vehicle Stereo Event Camera Dataset): A real-world dataset containing outdoor driving and indoor flight sequences with ground truth optical flow derived from LiDAR depth maps and IMU data.
  • HACD (HVGA ATIS Corner Dataset): A dataset recorded with planar patterns to evaluate corner detectors, providing ground truth optical flow.

Key Insights and Ideas

  • Time Surface as a Rich Representation: The time surface effectively captures motion information by encoding the temporal dynamics of events at each pixel, serving as a direct input for optical flow estimation.
  • Timestamp Consistency is Key: The core insight is that by ensuring temporal consistency between the current and a slightly shifted time surface, one can accurately infer motion without needing explicit intensity information.
  • Stable Loss Landscape: The proposed Surface Matching Loss exhibits a gentler and more stable loss landscape compared to variance-based losses, making optimization more robust and less prone to getting stuck in local minima.
  • Accurate Gradients: The gradients derived from the Surface Matching Loss are more accurate, especially around discontinuities, unlike variance-based methods where gradients can be erroneous at edges.
  • Direct Optical Flow Estimation: The method successfully estimates dense optical flow directly from events, bypassing the computationally expensive and potentially error-prone step of reconstructing image intensity.

Key Results

  • Improved Accuracy: The proposed Surface Matching method, when combined with L1 smoothness regularization, achieved higher accuracy than conventional methods like "Reconstruction" (simultaneous luminance restoration) and "Variance" (contrast maximization) on several sequences from the MVSEC and HACD datasets. Specifically, it was superior in two of five MVSEC sequences, all indoor flying sequences, and HACD.
  • Stable Loss Landscape: Visualizations of the loss landscape demonstrated that the Surface Matching Loss is wider and more stable across different textures compared to the narrow peaks of the variance loss.
  • Accurate Gradients: Experiments showed that the gradients of the Surface Matching Loss are more consistently calculated, even at edges, whereas variance-based methods can produce erroneous or reversed gradients.
  • Runtime: The method achieved approximately 3 fps on a CPU-based MATLAB implementation. The authors suggest that real-time implementation is feasible on a GPU, similar to image-based TV-L1 optical flow, due to the shared optimization framework.

Limitations and Future Work

  • Hyperparameter Sensitivity: The effective time of the time surface ($\tau$) and the duration of the optical flow ($\Delta t$) are important hyperparameters that need to be adjusted for each scene. The paper investigated their impact, finding that shorter $\Delta t$ generally leads to higher accuracy.
  • Future Work: The authors suggest developing a mechanism to adaptively determine the optimal $\Delta t$ parameter for each scene, which would further enhance the method's robustness and ease of use.

Summary updated

Abstract

In this work, we propose a novel method of estimating optical flow from event-based cameras by matching the time surface of events. The proposed loss function measures the timestamp consistency between the time surface formed by the latest timestamp of each pixel and the one that is slightly shifted in time. This makes it possible to estimate dense optical flows with high accuracy without restoring luminance or additional sensor information. In the experiment, we show that the gradient was more correct and the loss landscape was more stable than the variance loss in the motion compensation approach. In addition, we show that the optical flow can be estimated with high accuracy by optimization with L1 smoothness regularization using publicly available datasets.
BibTeX
@inproceedings{nagata2021optical,
  author = {Jun Nagata and Yusuke Sekikawa and Yoshimitsu Aoki},
  title = {{Optical Flow Estimation by Matching Time Surface with Event-Based Cameras}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=zk5w2PPYBA},
}
Figure