TL;DR
This paper introduces a novel "edge-informed" approach to contrast maximization for event-based vision, significantly improving optical flow estimation by jointly optimizing event contrast and edge correlation. This hybrid method achieves state-of-the-art results on challenging event-based datasets, demonstrating the power of integrating information from both event cameras and traditional image frames.
What Problem Does This Paper Address?
Event cameras are a fascinating new type of sensor that asynchronously detect changes in brightness, making them ideal for capturing fast motion. However, processing their data for tasks like optical flow estimation is challenging due to the sparse and temporally dense nature of events. A common technique, Contrast Maximization (CM), aims to estimate motion trajectories by warping events to maximize their "sharpness" or contrast. While effective, CM has limitations: it can sometimes lead to "event collapse" (where events warp to too few pixels) and its performance can be further improved. This paper addresses these limitations by proposing a way to leverage not just the events themselves, but also the structural information from traditional image frames (specifically, edges) to guide the CM process. This hybrid approach aims to overcome the inherent challenges of event-based optical flow and push the boundaries of its performance.
Methods
The core innovation of this paper is the development of an Edge-Informed Contrast Maximization (EINCM) framework. This builds upon existing Contrast Maximization (CM) techniques, which optimize motion parameters to maximize the contrast of a warped event image (IWE). The key additions are:
- Bi-modal Objective: Instead of solely relying on event contrast, EINCM introduces a second objective: event-edge correlation. This measures how well the warped events align with the edges detected in traditional image frames. This is achieved by extracting edges from image frames (using a pipeline involving denoising, CLAHE, sharpening, and Canny edge detection) and then calculating the Mean Squared Error (MSE) between the IWE and the edge image. The negative MSE is used as the correlation objective.
- Hybrid Objective Function: The overall optimization objective is a weighted sum of the relative event contrast and the relative event-edge correlation. This allows the framework to benefit from both the temporal information in events and the spatial structure from image frames.
- Multiscale and Multireference Techniques: The paper refines and adapts existing multiscale and multireference strategies (used in previous CM methods) for this bi-modal setting. This involves processing data at different pyramid levels and using multiple reference times to ensure robust convergence and prevent overfitting.
- Pre-handover Multiscale Strategy: A novel "pre-handover" multiscale strategy is introduced, where optimization and handover operations occur before upsampling at each pyramid level. This is contrasted with the "post-handover" approach used in prior work and is shown to yield superior performance.
- Handover Weight Optimization: For the handover operation (combining information from different pyramid levels), the paper explores both fixed and learned weights, finding that a hybrid approach (solved weights on finer levels, fixed on coarser) performs best.
Datasets
The paper evaluates its method on three publicly available event-based datasets:
- MVSEC: Contains stereo sequences of indoor and outdoor environments. Ground truth optical flow is derived from camera velocity and scene depth, but can have synchronization issues and small displacements.
- DSEC: Features driving scenes with larger displacements. However, events and images are captured by different cameras, leading to potential registration challenges.
- ECD: Provides a variety of scenes but lacks ground-truth optical flow, making it useful for evaluating qualitative aspects and sharpness metrics.
The authors do not introduce a new dataset but highlight the challenges and differences of these existing ones, particularly regarding ground-truth accuracy and sensor synchronization.
Key Insights and Ideas
- Edges are Crucial for Event-Based Vision: The paper strongly suggests that incorporating edge information from traditional frames is not just an enhancement but a fundamental step towards more accurate and robust event-based optical flow. Events themselves are generated by intensity gradients (edges), so aligning warped events with detected edges provides a powerful, complementary cue.
- Bi-modal Fusion Outperforms Uni-modal: By combining event contrast with edge correlation, the EINCM method achieves significantly better results than event-only CM approaches, even outperforming state-of-the-art learning-based methods in some aspects.
- Pre-handover is Better than Post-handover: The novel pre-handover multiscale strategy proves to be more effective than the post-handover approach used in prior work, leading to superior performance. This suggests that optimizing at a coarser level before upsampling and passing information is more beneficial.
- Handover Strategy Matters: The way information is passed between pyramid levels (handover) significantly impacts performance. The paper demonstrates that a carefully chosen strategy, involving solving for handover weights at finer scales, is crucial for optimal results.
- Model-Based Methods Can Compete with Learning-Based: Despite the rise of deep learning, this work shows that well-designed model-based approaches, especially when incorporating multi-modal information, can achieve competitive or even superior results without requiring large amounts of labeled training data.
Key Results
- State-of-the-Art Performance: EINCM establishes new state-of-the-art benchmarks on the MVSEC, DSEC, and ECD datasets for event-based optical flow, particularly for model-based methods.
- Improved Sharpness (FWL): The paper reports significantly better Flow Warp Loss (FWL) scores, indicating sharper and more accurate warped event images, which is a key metric for event-based vision. For example, on MVSEC (dt=4), EINCM achieves an FWL of 1.439 compared to 1.68 for the previous state-of-the-art [30].
- Competitive Accuracy (AEE): While not always outperforming supervised learning methods on accuracy metrics like Average Endpoint Error (AEE), EINCM shows strong competitive performance, especially given its model-based nature. On DSEC, EINCM achieves AEE scores around 2-3, comparable to other model-based methods.
- Event-Only Baseline: Even without using edge information, the paper's event-only CM baseline outperforms previous state-of-the-art event-only methods on MVSEC.
Limitations and Future Work
- Reliance on Frame Data: The EINCM method's performance is dependent on the quality of edge extraction from image frames. This means that low-quality images, poor lighting, or inaccurate registration between event cameras and image sensors can hinder performance.
- Aperture Problem: Like many optical flow methods, this approach does not directly solve the aperture problem, which can lead to difficulties in areas with ambiguous motion.
- Noise Sensitivity: Extremely noisy event data or situations where events are not solely due to motion might pose challenges.
Future work suggested by the authors includes:
- Enhancing Edge Extraction: Developing more robust edge extraction techniques that are less sensitive to image quality and registration issues.
- Practical Integration: Exploring more seamless integration of event cameras with conventional cameras in real-world applications.
- Addressing Aperture Problem: Investigating methods to mitigate the limitations imposed by the aperture problem.
Summary updated