Abstract: Real-world scenes have a dynamic range of up to 280 dB that today's imaging sensors cannot directly capture. Existing live vision pipelines tackle this fundamental challenge by relying on high dynamic range (HDR) sensors that try to recover HDR images from multiple captures with different exposures. While HDR sensors substantially increase the dynamic range, they are not without disadvantages, including severe artifacts for dynamic scenes, reduced fill-factor, lower resolution, and high sensor cost. At the same time, traditional auto-exposure methods for low-dynamic range sensors have advanced as proprietary methods relying on image statistics separated from downstream vision algorithms. In this work, we revisit auto-exposure control as an alternative to HDR sensors. We propose a neural network for exposure selection that is trained jointly, end-to-end with an object detector and an image signal processing (ISP) pipeline. To this end, we use an HDR dataset for automotive object detection and an HDR training procedure. We validate that the proposed neural auto-exposure control, which is tailored to object detection, outperforms conventional auto-exposure methods by more than 6 points in mean average precision (mAP).
Introduction
Real-world lighting is incredibly diverse, spanning a dynamic range of up to 280 dB—from the dim light of starlight to the intense brightness of direct sunlight. However, standard CMOS sensors typically capture only about 60-70 dB in a single shot. To bridge this gap, the industry often turns to expensive High-Dynamic Range (HDR) sensors that merge multiple exposures. While effective, these sensors are prone to "ghosting" artifacts in dynamic scenes and come with significant hardware costs.
Alternatively, cameras use Auto-Exposure (AE) to select which portion of the light range to capture. Traditionally, AE is a "black box" designed by manufacturers to produce aesthetically pleasing images for human viewers. However, what looks "good" to a person is not necessarily optimal for a computer vision algorithm. For instance, a traditional AE might overexpose a car exiting a dark tunnel to make the sky look natural, causing the detector to miss the vehicle entirely. This paper proposes a shift: training the camera to "see" specifically for the algorithms that consume the data.
System Architecture and Methods
The core of the proposed approach is a Hybrid Neural Network for exposure control that is trained end-to-end with an object detector. This system moves away from aesthetic-driven exposure toward task-specific control.
The exposure controller consists of two main branches:
- Histogram NN: This branch analyzes global image statistics using multi-scale histograms. It is computationally efficient and captures the overall lighting distribution of the scene.
- Semantic NN: This branch reuses features from the object detector’s own backbone (specifically, the output of a ResNet
conv2layer). This allows the exposure controller to "understand" the semantic content of the scene and prioritize the visibility of relevant objects, such as pedestrians or vehicles in shadows.
To enable joint optimization, the researchers utilized a differentiable Image Signal Processor (ISP). This allows mathematical gradients to flow from the final object detection loss back to the exposure settings. Since training on a live camera in a loop is impractical, the process was simulated by taking high-quality HDR data and mathematically "downgrading" it into various Low-Dynamic Range (LDR) captures with different simulated exposures and noise levels.
Experimental Setup
The study utilized a custom Automotive HDR Dataset consisting of 1,600 image pairs captured with a high-end Sony IMX490 HDR sensor (140 dB). The dataset covers a wide variety of conditions, including daytime, dusk, and night in urban, suburban, and highway environments. The second frame of each pair was manually annotated for 2D object detection to provide ground truth for training the detector.
Key Results and Insights
The most significant takeaway is that the camera's "front-end" (exposure) should not be decoupled from the "back-end" (the vision task). By tailoring exposure to detection, the system learns to ignore irrelevant bright spots, like the sun, to focus on relevant dark areas, like a pedestrian in the shadows.
In simulations with large exposure shifts, the Hybrid NN outperformed standard "Average AE" and "Gradient AE" by over 6 points in mean Average Precision (mAP). Real-world validation on a prototype vehicle confirmed these findings; in tricky scenarios like tunnel exits, the neural AE system achieved 32.37 mAP compared to the baseline's 28.80 mAP. Qualitative results demonstrate that the system is significantly better at balancing exposure to keep objects visible in both high-contrast shadows and bright highlights simultaneously.
Conclusion
This approach suggests that HDR challenges can be addressed through intelligent software control rather than relying solely on expensive hardware. Future work may involve more advanced temporal modeling (e.g., LSTMs or Transformers) to predict lighting changes and the integration of this logic into low-power ISP hardware. Ultimately, this research points toward a future of "joint optimization," where physical sensor architectures and neural controllers are designed together from the ground up.
Summary updated