End-to-End High Dynamic Range Camera Pipeline Optimization

Nicolas Robidoux, Luis E. Garcia Capel, Dongeun Seo, Avinash Sharma, Federico Ariza, Felix HeideCVPR 2021 · OpenReview

Abstract

With a 280 dB dynamic range, the real world is a High Dynamic Range (HDR) world. Today's sensors cannot record this dynamic range in a single shot. Instead, HDR cameras acquire multiple measurements with different exposures, gains and photodiodes, from which an Image Signal Processor (ISP) reconstructs an HDR image. HDR image recovery for dynamic scenes is an open challenge because of motion and because stitched captures have different noise characteristics, resulting in artefacts that the ISP has to resolve---in real time and at triple-digit megapixel resolutions. Traditionally, hardware ISP settings used by downstream vision modules have been chosen by domain experts. Such frozen camera designs are then used for training data acquisition and supervised learning of downstream vision modules. We depart from this paradigm and formulate HDR ISP hyperparameter search as an end-to-end optimization problem. We propose a mixed 0th and 1st-order block coordinate descent optimizer to jointly learn ISP and detector network weights using RAW image data augmented with emulated SNR transition region artefacts. We assess the proposed method for human vision and image understanding. For automotive object detection, the method improves mAP and mAR by 33% compared to expert-tuning and by 22% compared to recent state-of-the-art. The method is validated in an HDR laboratory rig and in the field, outperforming conventional handcrafted HDR imaging and vision pipelines in all experiments.

Overview

The real world presents a massive dynamic range of approximately 280 dB, far exceeding the capabilities of standard camera sensors. To capture this, HDR cameras multiplex data through multiple exposures or gains, which are then reconstructed by an Image Signal Processor (ISP). This paper introduces a method to jointly optimize the sensor, hardware ISP, and downstream vision models (e.g., CNNs) end-to-end. By replacing manual "expert tuning" with a hybrid optimization strategy, the authors achieve significant gains in object detection accuracy for automotive HDR scenes.

The HDR Challenge: From Real World to Display

Standard sensors capture only a fraction of the real world's dynamic range in a single exposure. HDR imaging compensates by stitching multiple captures, but this process is inherently difficult. It often introduces artifacts like "ghosting" from motion and "SNR discontinuities"—visible jumps in noise levels where different exposures are merged.

The HDR camera pipeline: mapping the 280 dB real world through optics, sensor, and ISP to a low-dynamic-range display.

Traditionally, ISPs are hand-tuned by experts to produce aesthetically pleasing images. However, these "frozen" designs may discard information critical for computer vision tasks. Manual tuning also struggles to account for the complex noise characteristics inherent in HDR stitching.

End-to-End Optimization Framework

The authors propose a framework that treats the sensor, ISP, and CNN as a single, optimizable system.

Hybrid Optimization Strategy

Because hardware ISPs are typically non-differentiable "black boxes," they cannot be optimized using standard backpropagation. The authors employ a Block Coordinate Descent approach:

  • 0th-order Optimization: A variant of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) searches for optimal sensor and ISP hyperparameters.
  • 1st-order Optimization: Stochastic Gradient Descent (SGD) updates the weights of the downstream detector (YOLOv4).

Robustness through Artifact Emulation

To ensure the system can handle real-world imperfections without requiring exhaustive datasets, the authors developed a method to augment RAW data with simulated SNR transition artifacts. This allows the model to learn robustness against the noise "jumps" common in HDR pipelines.

Examples of HDR artifacts: (a) motion ghosting, (b) flare/reflections, and (c) SNR transition noise discontinuities.

Experimental Validation

The method was tested using both a custom HDR laboratory rig (covering a 123 dB range) and real-world automotive field data captured with a Sony IMX490 sensor.

Key Results

  • Object Detection: The proposed method improved mean Average Precision (mAP) and mean Average Recall (mAR) by 33% compared to expert-tuned pipelines and by 22% over previous state-of-the-art optimization methods.
  • Human Vision: For viewing tasks, the method outperformed expert tuning in preserving details across extreme luminance levels (ranging from 0.222 to 312,000 $cd/m^2$).
  • Efficiency: The joint optimization successfully managed hundreds of ISP hyperparameters, finding stable solutions that consistently outperformed manual tuning.

Insights and Future Directions

A primary takeaway is the "Pretty Image Trap": ISP settings optimized for human aesthetics often perform poorly for computer vision. Joint optimization allows the ISP to preserve low-bit information—such as shadow details—that a human might perceive as "noisy" but a CNN finds essential for accurate detection.

While highly effective, the current pipeline focuses on single-frame processing. Future work could extend this to temporal optimization, accounting for video-specific features like 3D noise reduction, and explore generalization across other tasks like semantic segmentation or depth estimation.

Summary updated

Abstract

With a 280 dB dynamic range, the real world is a High Dynamic Range (HDR) world. Today's sensors cannot record this dynamic range in a single shot. Instead, HDR cameras acquire multiple measurements with different exposures, gains and photodiodes, from which an Image Signal Processor (ISP) reconstructs an HDR image. HDR image recovery for dynamic scenes is an open challenge because of motion and because stitched captures have different noise characteristics, resulting in artefacts that the ISP has to resolve---in real time and at triple-digit megapixel resolutions. Traditionally, hardware ISP settings used by downstream vision modules have been chosen by domain experts. Such frozen camera designs are then used for training data acquisition and supervised learning of downstream vision modules. We depart from this paradigm and formulate HDR ISP hyperparameter search as an end-to-end optimization problem. We propose a mixed 0th and 1st-order block coordinate descent optimizer to jointly learn ISP and detector network weights using RAW image data augmented with emulated SNR transition region artefacts. We assess the proposed method for human vision and image understanding. For automotive object detection, the method improves mAP and mAR by 33% compared to expert-tuning and by 22% compared to recent state-of-the-art. The method is validated in an HDR laboratory rig and in the field, outperforming conventional handcrafted HDR imaging and vision pipelines in all experiments.
BibTeX
@inproceedings{robidoux2021end,
  author = {Nicolas Robidoux and Luis E. Garcia Capel and Dongeun Seo and Avinash Sharma and Federico Ariza and Felix Heide},
  title = {{End-to-End High Dynamic Range Camera Pipeline Optimization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=rD3cYsAaBh},
}
Figure