Toward RAW Object Detection: A New Benchmark and A New Model

Ruikang Xu, Chang Chen, Jingyang Peng, Cheng Li, Yibin Huang, Fenglong Song, Youliang Yan, Zhiwei XiongCVPR 2023 · OpenReview

Abstract

In many computer vision applications (e.g., robotics and autonomous driving), high dynamic range (HDR) data is necessary for object detection algorithms to handle a variety of lighting conditions, such as strong glare. In this paper, we aim to achieve object detection on RAW sensor data, which naturally saves the HDR information from image sensors without extra equipment costs. We build a novel RAW sensor dataset, named ROD, for Deep Neural Networks (DNNs)-based object detection algorithms to be applied to HDR data. The ROD dataset contains a large amount of annotated instances of day and night driving scenes in 24-bit dynamic range. Based on the dataset, we first investigate the impact of dynamic range for DNNs-based detectors and demonstrate the importance of dynamic range adjustment for detection on RAW sensor data. Then, we propose a simple and effective adjustment method for object detection on HDR RAW sensor data, which is image adaptive and jointly optimized with the downstream detector in an end-to-end scheme. Extensive experiments demonstrate that the performance of detection on RAW sensor data is significantly superior to standard dynamic range (SDR) data in different situations. Moreover, we analyze the influence of texture information and pixel distribution of input data on the performance of the DNNs-based detector. Code and dataset will be available at https://gitee.com//mindspore/models/tree/master/research/cv/RAOD.

Introduction

Standard object detection relies on 8-bit Standard Dynamic Range (SDR) images processed by an Image Signal Processor (ISP). While these images are visually pleasing to humans, the ISP pipeline is inherently lossy. In high-contrast scenes—such as driving directly into a sunset or navigating dark streets with bright oncoming headlights—the ISP often clips highlights or crushes shadows. This process discards critical high-dynamic-range (HDR) information that autonomous systems need for safety-critical decision-making.

Extreme glare conditions in the ROD dataset, where standard ISP processing often fails to preserve object details.

While RAW data preserves the sensor's full dynamic range, it presents significant challenges: it is uncompressed, lacks color consistency, and its pixel distribution is heavily skewed. Existing RAW datasets are often too small or limited to 12-14 bits, which is insufficient for extreme real-world lighting conditions.

Proposed Method: Deep Image-Adaptive Processing

To address these challenges, we propose an end-to-end pipeline that replaces the traditional ISP with a Deep Image-Adaptive Processing network. This lightweight module is jointly trained with a downstream detector (e.g., YOLOX or Sparse R-CNN) and consists of two primary components:

  1. Image-Level Adjustment: This module learns a global transformation (similar to a learnable gamma correction, $X^\gamma$) to adaptively shift the pixel distribution based on global image characteristics.
  2. Pixel-Level Adjustment: To preserve local details, this module employs a piecewise linear transformation. It segments RAW data by intensity and applies local masks to enhance texture and contrast in specific regions.

The outputs are fused and passed to the detector. The entire system is optimized for machine accuracy rather than human aesthetics, using only detection loss (classification and bounding box regression).

The ROD Dataset

We introduce the ROD (RAW Object Detection) dataset, the first large-scale 24-bit RAW sensor dataset for object detection:

  • Scale: 25,000 annotated RAW images.
  • Dynamic Range: 24-bit, captured using a Sony IMX490 sensor that combines sub-pixels with different exposure times.
  • Content: 237,000 instances across 6 classes (Car, Pedestrian, Cyclist, Tram, Tricycle, Track) in diverse day and night driving scenes.

Experimental Results and Insights

Our experiments reveal a "RAW Paradox": simply feeding high-bit RAW data into standard detectors often yields worse results than 8-bit SDR images because useful information is buried in a tiny fraction of the 24-bit space. We found that Dynamic Range Adjustment (DRA) is the most critical component for overcoming this.

Qualitative detection results across various challenging scenarios in the ROD dataset, including day, night, and high-contrast environments.

Key performance metrics include:

  • Superior Accuracy: On the ROD dataset, our model achieved 58.7 AP (daytime) and 54.2 AP (nighttime), significantly outperforming SDR baselines (52.1 AP and 50.3 AP, respectively).
  • Efficiency: The adjustment module is extremely lightweight, adding only 0.08M parameters and 0.64 GFLOPs, making it suitable for real-time autonomous driving.
  • Robustness: Qualitative analysis shows the model successfully detects vehicles in extreme glare where SDR-based models fail completely.

Conclusion and Future Work

This work demonstrates that RAW-domain detection significantly outperforms traditional SDR approaches in challenging lighting. While the current model requires some initial down-sampling, it paves the way for "RAW-native" architectures. Future research will focus on developing networks that can exploit HDR information directly without needing an explicit adjustment module.

Summary updated

Abstract

In many computer vision applications (e.g., robotics and autonomous driving), high dynamic range (HDR) data is necessary for object detection algorithms to handle a variety of lighting conditions, such as strong glare. In this paper, we aim to achieve object detection on RAW sensor data, which naturally saves the HDR information from image sensors without extra equipment costs. We build a novel RAW sensor dataset, named ROD, for Deep Neural Networks (DNNs)-based object detection algorithms to be applied to HDR data. The ROD dataset contains a large amount of annotated instances of day and night driving scenes in 24-bit dynamic range. Based on the dataset, we first investigate the impact of dynamic range for DNNs-based detectors and demonstrate the importance of dynamic range adjustment for detection on RAW sensor data. Then, we propose a simple and effective adjustment method for object detection on HDR RAW sensor data, which is image adaptive and jointly optimized with the downstream detector in an end-to-end scheme. Extensive experiments demonstrate that the performance of detection on RAW sensor data is significantly superior to standard dynamic range (SDR) data in different situations. Moreover, we analyze the influence of texture information and pixel distribution of input data on the performance of the DNNs-based detector. Code and dataset will be available at https://gitee.com//mindspore/models/tree/master/research/cv/RAOD.
BibTeX
@inproceedings{xu2023toward,
  author = {Ruikang Xu and Chang Chen and Jingyang Peng and Cheng Li and Yibin Huang and Fenglong Song and Youliang Yan and Zhiwei Xiong},
  title = {{Toward RAW Object Detection: A New Benchmark and A New Model}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=cOEEjApFEV},
}
Figure