TL;DR
This paper introduces the Stereo Hybrid Event-Frame (SHEF) camera system, which combines separate, high-quality event and frame cameras to overcome the limitations of existing hybrid sensors. They propose a novel stereo disparity estimation method that leverages both event and frame data, achieving state-of-the-art performance on their newly introduced SHEF dataset.
What Problem Does This Paper Address?
Traditional cameras struggle with challenging visual conditions like low light and fast motion due to their frame-based nature, leading to motion blur and limited dynamic range. Event cameras, which detect pixel-level brightness changes asynchronously, offer solutions to these issues with high dynamic range, low latency, and minimal motion blur. However, they lack absolute intensity information. Existing hybrid event-frame cameras (like DAVIS) attempt to combine both, but their integrated pixel design compromises data quality. This paper addresses the need for a robust stereo perception system that can leverage the strengths of both event and frame sensing modalities without compromise, particularly for near-field 3D perception tasks in robotics.
Methods
The core of the paper is the Stereo Hybrid Event-Frame (SHEF) camera system. Unlike integrated hybrid sensors, SHEF uses two distinct, high-quality cameras: a Prophesee event camera and a FLIR RGB frame camera, mounted side-by-side. This separation ensures that each sensor's data is uncompromised.
For 3D perception, they tackle the stereo disparity estimation problem. Their approach involves:
- Binary Edge Extraction: High-pass filtering is applied to event data to reconstruct intensity images, from which binary edge maps are generated. Similarly, Sobel filters are used on frame data to create corresponding edge maps.
- Cross-Correlation for Sparse Disparity: A coarse-to-fine framework is employed. Initially, a sparse disparity map ($D_s$) is computed by sliding a window along epipolar lines and calculating cross-correlation between the reference edge map from one camera and the edge map from the other.
- Coarse-to-Fine Pyramid: To densify and smooth the disparity map, a 5-level image pyramid with a scale factor of 0.9 is used. Disparity estimation is performed iteratively across these pyramid levels, starting from the coarsest. This results in a denser disparity map ($D_p$).
- Disparity Completion Network (DCNet): To further refine the disparity map and generate a dense output, they train a DCNet. This network takes the sparse disparity map ($D_p$), the frame image ($L$), and the event stream ($E_s$) as input to produce a final dense disparity map ($D_e$).
Datasets
The paper introduces a newly created SHEF dataset specifically for stereo hybrid event-frame depth estimation in near-field scenes. This dataset includes:
- Sensors: Prophesee VGA event sensor and FLIR Chameleon3 USB3 RGB camera (640x480 and 2048x1536 resolution, respectively).
- Environments: Three distinct scenes: "simple boxes," "complex boxes," and "picnic."
- Camera Motion: Controlled by a UR5 robot manipulator, allowing for precise and reproducible trajectories (circles, '8' shapes, translations) at various speeds and lighting conditions.
- Ground Truth: Dense depth ground truth for each frame, generated from 3D reconstruction models created using commercial software (3DFLOW Zephyr) and accurate camera poses.
- Data: Includes event data, RGB frames (54 fps), and camera trajectories (approx. 125 Hz).
For training their DCNet, they also mention using the Middlebury dataset, which provides stereo image pairs, camera parameters, and ground truth disparity maps.
Key Insights and Ideas
- Separation is Key: The fundamental insight is that using separate, high-quality event and frame cameras in a stereo setup offers superior performance compared to integrated hybrid sensors where pixel-level coupling can degrade data quality.
- Event-Frame Edge Correlation: Leveraging the complementary nature of event and frame data by correlating their respective edge information is a powerful strategy for disparity estimation, especially in challenging scenarios where one modality might be noisy or incomplete.
- Coarse-to-Fine for Robustness: The coarse-to-fine framework significantly improves the density and smoothness of the disparity map by averaging out noise inherent in event data and leveraging spatial information.
- Dataset Importance: The creation of a dedicated SHEF dataset is crucial for advancing research in this specific area, as existing datasets may not fully capture the nuances of separate event and frame sensor fusion for stereo perception.
Key Results
- Baseline Performance: Their proposed baseline disparity estimation algorithm, using cross-correlation and a coarse-to-fine framework, significantly outperforms the state-of-the-art stereo frame-based matching algorithm AANet on their SHEF dataset.
- Quantitative Improvements:
- The coarse-to-fine disparity map ($D_p$) achieved over 51% relative improvement in RMSE and 55% in "bad-p" compared to the sparse disparity map ($D_s$).
- $D_p$ outperformed AANet (evaluated on edge pixels) with a significant margin in all three metrics (RMSE, bad-p, $\delta_1$, $\delta_2$, $\delta_3$).
- The dense disparity completion using DCNet ($D_e$) achieved an RMSE of 8.99 and a "bad-p" of 31.71%, outperforming AANet's dense disparity (RMSE 10.66, "bad-p" 56.61%) on the SHEF dataset.
- Qualitative Results: Visualizations show that their method produces smoother and more accurate disparity maps, particularly in challenging areas where AANet struggles. The 3D reconstructions from their disparity maps are sufficient for robotic tasks like grasping.
Limitations and Future Work
- Limitations:
- The paper acknowledges that event camera edges can be noisy and highly dependent on camera motion, which can affect the initial disparity estimation ($D_s$).
- While their DCNet is trained on their dataset, it's noted that AANet and DCNet were fine-tuned or trained on their generated training data for fair comparison.
- The dataset, while comprehensive, is focused on near-field scenes.
- Future Work:
- The authors suggest that their SHEF system and dataset can be used for further research in stereo disparity estimation and other 3D perception tasks.
- The DCNet approach demonstrates the potential for better visualization and combining sparse disparity with dense images, hinting at further exploration in this area.
- The paper's contribution of a robust baseline and dataset lays the groundwork for developing more advanced algorithms for hybrid stereo perception.
Summary updated