TL;DR
This paper introduces "Event Spectroscopy," a novel perception system that uses a single event camera and a structured light projector to simultaneously capture high-resolution depth and multispectral data. By exploiting the hardware quirks of event sensors, the system accurately estimates scene reflectivity and depth with low latency, offering a lightweight, all-in-one alternative to carrying separate depth, RGB, and multispectral cameras on UAVs.
What Problem Does This Paper Address?
Uncrewed aerial vehicles (UAVs) are increasingly used in forests for environmental monitoring and search-and-rescue. To navigate safely and collect useful data, these drones must dodge thin, fast-moving branches and differentiate between foliage and woody structures.
Currently, this requires a bulky payload of multiple sensors: a depth camera for obstacle avoidance, an RGB camera for context, and a multispectral camera for assessing plant health. Furthermore, traditional sensors struggle in forests. Passive cameras suffer from the extreme lighting variations under the canopy, and standard depth cameras (like time-of-flight or active stereo) often fail to resolve thin structures like twigs or suffer from high latency. The authors tackle this by creating a single, unified sensor setup capable of high-speed, high-resolution depth and spectral imaging.
Methods
The system pairs an event camera with an active illumination source (a projector).
- Depth Estimation: The system uses event-based structured light. A projector rapidly scans a point of light across the scene. The event camera, which detects changes in illumination with microsecond latency, registers the flashes. Standard stereo triangulation between the projector and camera yields a high-resolution depth map.
- Multispectral Sensing: Event cameras theoretically only measure changes in light, not absolute intensity. However, the authors exploit a hardware non-ideality: the bandwidth of the pixel's source-follower circuit depends on absolute light intensity. Pixels looking at darker regions respond more slowly to the projector's flash. By varying the camera's contrast threshold and observing event generation, the system estimates absolute reflectivity.
- Hardware Setups: The authors built a lab setup using a full-spectrum light source with optical filters and a chopper, as well as a portable prototype using a commercial RGB laser projector.
Key Results
The proposed system demonstrates significant improvements over existing commercial solutions in both depth accuracy and spectral reconstruction.
- Depth Accuracy: The system achieved up to a 60% improvement in RMSE and up to a 125% improvement in Chamfer distance compared to commercial depth sensors like the Kinect V2 and Intel RealSense D435. It was particularly superior at capturing thin structures like wireframes, which often disappear in standard depth maps.
- Color and Spectral Accuracy: The method improved color accuracy (RMSE) by over 50% compared to existing event-counting baselines. Reconstructed spectral response curves (650 nm to 850 nm) closely matched ground-truth spectrometer data.
- Material Differentiation: In real-world rainforest tests, fusing event-based depth with RGB data improved the Intersection-over-Union (IoU) for "leaves" vs. "branches" segmentation by over 30% compared to using RGB alone.
Key Insights
- Turning Limitations into Features: The authors turn a known hardware limitation—illumination-dependent latency—into a functional feature for extracting absolute scene reflectivity.
- Perfect Registration: Because depth and spectral data come from the same event stream, the resulting data is perfectly aligned without complex extrinsic calibration.
- Lighting Robustness: Active illumination and the high dynamic range of event cameras allow the system to perform reliably across extreme lighting conditions.
Limitations and Future Work
The portable prototype is currently restricted to RGB due to commercial projector limitations. True portable multispectral sensing requires cycling through wavelengths, which increases capture time. Additionally, strong ambient sunlight can wash out the projector's signal at long distances. Future work will explore simultaneous multi-wavelength capture and higher-power light sources for outdoor reliability.
Summary updated