Neural Exposure Fusion for High-Dynamic Range Object Detection
Introduction
In real-world environments, lighting conditions vary dramatically, from the deep shadows of a tunnel to the blinding glare of direct sunlight. While the human visual system adapts seamlessly to these extremes, standard CMOS sensors are limited by a dynamic range of approximately 60-70 dB. To overcome this, High-Dynamic Range (HDR) imaging typically captures multiple bracketed exposures. However, traditional methods of fusing these exposures are often optimized for human perception rather than machine vision. This paper introduces Neural Exposure Fusion, a novel approach that merges exposures in the feature domain using a local cross-attention mechanism, significantly enhancing object detection in challenging lighting.
The Challenge of HDR for Machine Vision
The standard approach to HDR involves an Image Signal Processor (ISP) that fuses multiple captures into a single HDR image, which is then "tonemapped" for display. While this produces visually pleasing results, it is often detrimental to computer vision tasks. Tonemapping tends to smooth out noise and transitions, inadvertently destroying subtle gradients or high-frequency details that an object detector relies on to identify a pedestrian in a dark shadow or a traffic sign in a bright glare. In essence, perceptual beauty is often achieved at the cost of machine task performance.
Neural Exposure Fusion Pipeline
The authors propose a "task-driven" fusion pipeline that moves the merging step from pixels to neural features, allowing the model to retain information that would otherwise be lost in a traditional ISP.
The pipeline consists of four primary stages:
- Multi-Exposure Input: The system takes a stack of $n$ raw Low-Dynamic Range (LDR) exposures.
- Differentiable ISP: Each exposure passes through a shared, differentiable ISP and a lightweight ResNet feature extractor. Because the ISP is differentiable, the entire system can be trained end-to-end using a detection loss.
- Local Cross-Attention Fusion: This is the core innovation. Instead of a simple weighted average of pixels, the model uses an attention mechanism. At every spatial location, the model learns a "query" that looks across the different exposure features ("keys" and "values") to determine which exposure provides the most useful information for that specific spot.
- Detection Head: The fused feature map is then fed into a standard detection head (Faster R-CNN) to produce final bounding boxes.
Crucially, the attention mechanism is local, meaning it only compares features at the same $(x, y)$ coordinate across the different exposure layers. This design choice provides the benefits of attention-based selection without the massive memory and computational overhead associated with global Transformers.
Key Insights and Results
- Feature Fusion vs. Image Fusion: By fusing in the feature domain, the network can preserve "ugly" but useful information—such as sensor noise or extreme gradients—that a traditional ISP would filter out as artifacts.
- Intelligent Selection: Learned attention maps demonstrate the model's logic; for instance, in a tunnel exit scenario, the model automatically assigns high weights to short exposures for the bright exterior and long exposures for the dark interior.
- Accuracy and Robustness: Evaluated on a dataset of automotive HDR images (Sony IMX490 sensor), the proposed method outperformed standard HDR fusion baselines by 2.7% mAP. The gains were most significant for high-contrast categories like "Person" and "Traffic Sign."
- Real-Time Efficiency: The pipeline runs at 57 FPS on an NVIDIA Jetson Orin AGX using 16-bit float precision, making it viable for deployment on autonomous vehicles.
Conclusion
Neural Exposure Fusion represents a shift toward task-specific imaging pipelines. By leveraging local cross-attention in the feature domain, the system achieves superior object detection performance in extreme lighting conditions while maintaining the speed required for real-time robotics. Future research may explore the benefits of this architecture for other vision tasks, such as semantic segmentation or optical flow, and move toward end-to-end sensor-to-task co-design.
Summary updated