Neural Auto-Exposure for High-Dynamic Range Object Detection

Emmanuel Onzon, Fahim Mannan, Felix HeideCVPR 2021 · OpenReview

Abstract: Real-world scenes have a dynamic range of up to 280 dB that today's imaging sensors cannot directly capture. Existing live vision pipelines tackle this fundamental challenge by relying on high dynamic range (HDR) sensors that try to recover HDR images from multiple captures with different exposures. While HDR sensors substantially increase the dynamic range, they are not without disadvantages, including severe artifacts for dynamic scenes, reduced fill-factor, lower resolution, and high sensor cost. At the same time, traditional auto-exposure methods for low-dynamic range sensors have advanced as proprietary methods relying on image statistics separated from downstream vision algorithms. In this work, we revisit auto-exposure control as an alternative to HDR sensors. We propose a neural network for exposure selection that is trained jointly, end-to-end with an object detector and an image signal processing (ISP) pipeline. To this end, we use an HDR dataset for automotive object detection and an HDR training procedure. We validate that the proposed neural auto-exposure control, which is tailored to object detection, outperforms conventional auto-exposure methods by more than 6 points in mean average precision (mAP).

Introduction

Real-world lighting is incredibly diverse, spanning a dynamic range of up to 280 dB—from the dim light of starlight to the intense brightness of direct sunlight. However, standard CMOS sensors typically capture only about 60-70 dB in a single shot. To bridge this gap, the industry often turns to expensive High-Dynamic Range (HDR) sensors that merge multiple exposures. While effective, these sensors are prone to "ghosting" artifacts in dynamic scenes and come with significant hardware costs.

Alternatively, cameras use Auto-Exposure (AE) to select which portion of the light range to capture. Traditionally, AE is a "black box" designed by manufacturers to produce aesthetically pleasing images for human viewers. However, what looks "good" to a person is not necessarily optimal for a computer vision algorithm. For instance, a traditional AE might overexpose a car exiting a dark tunnel to make the sky look natural, causing the detector to miss the vehicle entirely. This paper proposes a shift: training the camera to "see" specifically for the algorithms that consume the data.

System Architecture and Methods

The core of the proposed approach is a Hybrid Neural Network for exposure control that is trained end-to-end with an object detector. This system moves away from aesthetic-driven exposure toward task-specific control.

The proposed end-to-end architecture includes a differentiable ISP and an AEC loop that reuses semantic features from the object detector.

The exposure controller consists of two main branches:

  1. Histogram NN: This branch analyzes global image statistics using multi-scale histograms. It is computationally efficient and captures the overall lighting distribution of the scene.
  2. Semantic NN: This branch reuses features from the object detector’s own backbone (specifically, the output of a ResNet conv2 layer). This allows the exposure controller to "understand" the semantic content of the scene and prioritize the visibility of relevant objects, such as pedestrians or vehicles in shadows.

To enable joint optimization, the researchers utilized a differentiable Image Signal Processor (ISP). This allows mathematical gradients to flow from the final object detection loss back to the exposure settings. Since training on a live camera in a loop is impractical, the process was simulated by taking high-quality HDR data and mathematically "downgrading" it into various Low-Dynamic Range (LDR) captures with different simulated exposures and noise levels.

Experimental Setup

The study utilized a custom Automotive HDR Dataset consisting of 1,600 image pairs captured with a high-end Sony IMX490 HDR sensor (140 dB). The dataset covers a wide variety of conditions, including daytime, dusk, and night in urban, suburban, and highway environments. The second frame of each pair was manually annotated for 2D object detection to provide ground truth for training the detector.

Key Results and Insights

The most significant takeaway is that the camera's "front-end" (exposure) should not be decoupled from the "back-end" (the vision task). By tailoring exposure to detection, the system learns to ignore irrelevant bright spots, like the sun, to focus on relevant dark areas, like a pedestrian in the shadows.

Qualitative comparison of different AE methods across challenging lighting scenarios. The Hybrid NN maintains visibility for objects that traditional methods lose to over- or under-exposure.

In simulations with large exposure shifts, the Hybrid NN outperformed standard "Average AE" and "Gradient AE" by over 6 points in mean Average Precision (mAP). Real-world validation on a prototype vehicle confirmed these findings; in tricky scenarios like tunnel exits, the neural AE system achieved 32.37 mAP compared to the baseline's 28.80 mAP. Qualitative results demonstrate that the system is significantly better at balancing exposure to keep objects visible in both high-contrast shadows and bright highlights simultaneously.

Conclusion

This approach suggests that HDR challenges can be addressed through intelligent software control rather than relying solely on expensive hardware. Future work may involve more advanced temporal modeling (e.g., LSTMs or Transformers) to predict lighting changes and the integration of this logic into low-power ISP hardware. Ultimately, this research points toward a future of "joint optimization," where physical sensor architectures and neural controllers are designed together from the ground up.

Summary updated

Abstract

Real-world scenes have a dynamic range of up to 280 dB that today's imaging sensors cannot directly capture. Existing live vision pipelines tackle this fundamental challenge by relying on high dynamic range (HDR) sensors that try to recover HDR images from multiple captures with different exposures. While HDR sensors substantially increase the dynamic range, they are not without disadvantages, including severe artifacts for dynamic scenes, reduced fill-factor, lower resolution, and high sensor cost. At the same time, traditional auto-exposure methods for low-dynamic range sensors have advanced as proprietary methods relying on image statistics separated from downstream vision algorithms. In this work, we revisit auto-exposure control as an alternative to HDR sensors. We propose a neural network for exposure selection that is trained jointly, end-to-end with an object detector and an image signal processing (ISP) pipeline. To this end, we use an HDR dataset for automotive object detection and an HDR training procedure. We validate that the proposed neural auto-exposure control, which is tailored to object detection, outperforms conventional auto-exposure methods by more than 6 points in mean average precision (mAP).
BibTeX
@inproceedings{onzon2021neural,
  author = {Emmanuel Onzon and Fahim Mannan and Felix Heide},
  title = {{Neural Auto-Exposure for High-Dynamic Range Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=tZPg1hZOkc},
}
Figure