TL;DR
This paper presents a novel method to achieve full RGB and depth sensing using a standard monochrome event-based camera (EC) and a structured light projector. By projecting rapidly changing RGB patterns, the system reconstructs color and depth information, overcoming the limitations of monochrome ECs and the resolution trade-offs of current color ECs.
What Problem Does This Paper Address?
Event-based cameras (ECs) offer significant advantages like high dynamic range, low latency, and low power consumption, making them ideal for challenging environments and fast-moving scenarios. However, the first generation of ECs are monochrome, limiting their applicability in tasks requiring color information. While color ECs exist, they often suffer from reduced spatial resolution due to color filters. Furthermore, ECs inherently struggle with static or slowly moving scenes, as they only report changes in brightness. This paper addresses the challenge of obtaining rich, high-resolution color and depth information from a monochrome EC, even for static scenes, without compromising the EC's inherent advantages.
Methods
The core of the proposed method involves using a Digital Light Processing (DLP) projector to emit rapidly changing RGB patterns onto a scene. A monochrome event-based camera then captures the reflections of these patterns. The system synchronizes the projector and camera, allowing events to be tagged with the specific color of the projected light. By projecting red, green, and blue light sequentially, the EC can reconstruct color information.
Key aspects of the method include:
- Active Structured Light (ASL): The projector emits patterns of light that are dynamically adjusted. This allows for control over the event rate, preventing bus saturation on the EC.
- Adaptive Patterning: The system employs various structured light patterns (dots, lines, solid) with adjustable "coverage percentages" (CP) to balance detail, speed, and bandwidth. This allows for optimizing color detection based on the scene's dynamics.
- Color Reconstruction: By capturing events generated by red, green, and blue projected light, the system reconstructs color channels.
- Depth Sensing: The structured light patterns also enable depth estimation through triangulation, similar to traditional structured light techniques.
- White Balance and Color Correction: The paper discusses methods for adjusting the white balance of the projected light and performing color correction to achieve more natural-looking results.
Datasets
The paper does not explicitly mention training on specific datasets. Instead, it focuses on evaluating its method using a printed color wheel and a printed Macbeth color chart. The "ground truth" for comparison is obtained using a standard high-resolution frame-based camera.
Key Insights and Ideas
- Monochrome ECs can achieve color and depth: The most significant insight is that a monochrome EC, when paired with a structured light projector, can overcome its inherent color limitation and provide rich RGB-D information.
- Dynamic control of event rate is crucial: The paper highlights the importance of managing the event rate from the EC, especially when using an active projector. Their Adaptive Structured Light (ASL) approach provides a practical solution to prevent bus saturation.
- Trade-off between speed, detail, and bandwidth: The choice of structured light patterns and their coverage percentage directly impacts the trade-off between the speed of scanning, the level of detail captured, and the overall bandwidth required.
- Potential for high-resolution color without resolution loss: Unlike color ECs that use Color Filter Arrays (CFAs) and reduce resolution, this method reconstructs color information onto the full spatial resolution of the monochrome EC.
Key Results
- High equivalent frame rates: The system achieved color detection at an equivalent frame rate of up to 1400 fps, significantly faster than traditional frame-based RGB-D cameras.
- Improved color accuracy with white balance: Manual white balance tuning resulted in an average improvement of 12% in RMSE across color channels.
- Quantitative metrics: The paper reports RMSE (Root Mean Square Error) and Histogram Correlation (HC) values to quantify the quality of color detection, showing improved results with white balance calibration. For example, Histogram Correlation improved from 0.22 to 0.76 after white balance calibration.
- Qualitative results: Figures demonstrate the successful reconstruction of colorful images and boards, showcasing the method's ability to capture both static and moving objects with color and depth.
Limitations and Future Work
- Material limitations: The current method is primarily focused on matte, non-fluorescent materials and visual light wavelengths. It does not account for materials that absorb, scatter, or emit light differently.
- Assumptions in ASL: The Adaptive Structured Light (ASL) method makes simplifying assumptions about the power density of the DLP projector and the coverage percentage on the DLP frame plane, which might not hold true in all scenarios.
- Future work: The authors suggest investigating the use of different light sources and materials, as well as exploring more advanced shading models (like Blinn-Phong) for scanning reflective materials. They also mention the potential for using this method with color ECs and white light for static scenes.
Summary updated