ISP-Teacher: Image Signal Process with Disentanglement Regularization for Unsupervised Domain Adaptive Dark Object Detection

Yin Zhang, Yongqiang Zhang, Zian Zhang, Man Zhang, Rui Tian, Mingli DingAAAI 2024 · OpenReview

TL;DR

This paper introduces ISP-Teacher, a novel approach for dark object detection that tackles the limitations of current domain adaptation methods by explicitly modeling the image signal processing (ISP) pipeline. By using self-supervised learning to simulate realistic low-light image degradation and disentangling this process from the object detection task, ISP-Teacher significantly improves performance on challenging nighttime datasets.

What Problem Does This Paper Address?

Object detection systems perform admirably in well-lit conditions but struggle significantly in low-light or nighttime scenarios. This is a critical issue for applications like autonomous driving and security monitoring. Current solutions often rely on unsupervised domain adaptation (UDA) techniques, typically using a Teacher-Student architecture. These methods attempt to bridge the gap between daytime (source) and nighttime (target) data by applying data augmentation to daytime images to mimic nighttime conditions. However, the authors argue that these augmentation strategies are often non-learnable and fail to capture the complex, intrinsic image signal processing (ISP) that occurs within camera sensors when capturing low-light images. This neglect leads to domain bias and suboptimal performance, even falling below baseline detectors in some cases. The core problem is the failure to accurately model and adapt to the realistic degradation processes that occur in low-light image formation.

Methods

ISP-Teacher introduces two key innovations within a Teacher-Student UDA framework:

  • Self-supervised Learning based ISP Degradation (ISP-DTM): Instead of generic data augmentation, the authors propose a learnable module that mimics the camera's ISP pipeline. This "day-to-night transformation" (ISP-DTM) module takes daytime images and, through a series of simulated ISP steps (invert tone mapping, noise modeling, color transformation, gamma correction, signal quantization), generates realistic low-light images. Crucially, the parameters of this degradation process (like light intensity, gamma, and color gains) are learned in a self-supervised manner by comparing predicted degradation parameters to ground truth values derived from the ISP-DTM process itself. This allows the model to capture the intrinsic visual information that is robust to lighting changes.
  • Disentanglement Regularization (DR): A shared encoder is used for both the ISP degradation task and the object detection task. This can lead to "gradient conflict," where optimizing for one task negatively impacts the other. To address this, ISP-Teacher employs a disentanglement regularization technique. It aims to minimize the cosine similarity between the gradients of the ISP degradation task and the object detection task, effectively pushing them towards orthogonality. Simultaneously, it maximizes the cosine similarity of gradients within the same task. This ensures that the two tasks can be learned independently within the shared encoder, preventing interference and improving overall performance.

The overall loss function combines the self-supervised ISP degradation loss ($L_{self}$), the disentanglement regularization loss ($L_{DR}$), and the standard detection losses ($L_{det}$), weighted by a hyperparameter $\beta$.

Datasets

The paper evaluates ISP-Teacher on two widely used autonomous driving datasets:

  • BDD100k: This dataset contains a diverse range of driving scenarios, including various weather conditions. For this work, it was split into daytime images (source domain) and nighttime images (target domain).
  • SHIFT: This dataset is designed to capture domain shifts, including changes in weather and time of day. Similar to BDD100k, it was split into daytime (source) and nighttime (target) images.

The authors note that for fair comparison, they used the same backbone (ResNet50) and evaluation metrics (AP, APs, APM, APL) as previous state-of-the-art methods.

Key Insights and Ideas

  • ISP Modeling is Crucial: The paper strongly emphasizes that simply applying generic augmentations to mimic low-light conditions is insufficient. Accurately modeling the camera's internal image signal processing pipeline is key to generating realistic degradations and learning robust features.
  • Self-Supervision for Intrinsic Features: By learning the ISP degradation parameters in a self-supervised manner, the model can extract visual information that is inherently present in the scene, rather than being solely dependent on lighting conditions.
  • Gradient Orthogonality for Multi-Task Learning: The disentanglement regularization highlights the importance of managing gradient conflicts in multi-task learning. Pushing gradients towards orthogonality is an effective strategy to allow different tasks to coexist and benefit from a shared representation.
  • Teacher-Student Architectures Have Limitations in Extreme Domains: The paper demonstrates that even advanced Teacher-Student UDA methods struggle with the significant domain shift between day and night, often performing worse than simpler baselines.

Key Results

ISP-Teacher demonstrates significant improvements over existing methods:

  • BDD100k: Achieved an improvement of +2.4% AP over the state-of-the-art method (2PCNet), reaching 48.8% AP. It also outperformed the "Oracle" (supervised nighttime training) by +2.6% AP.
  • SHIFT: Achieved an improvement of +3.3% AP over the state-of-the-art method (2PCNet), reaching 52.4% AP. It also outperformed the "Oracle" by +5.4% AP.
  • Qualitative results show ISP-Teacher accurately detecting objects in challenging nighttime scenes where other methods missed objects or made incorrect detections.

Limitations and Future Work

  • Small Object Detection: The authors acknowledge that detecting small objects, such as traffic lights, still requires improvement.
  • Hyperparameter Sensitivity: The weight of the self-supervised learning loss ($\beta$) was found to be sensitive, with $\beta=1$ yielding the best results. Higher values led to performance degradation.
  • Future Work: The authors plan to explore Fourier-based mix strategies to learn more robust features for the student network, aiming to further improve performance, especially on small objects.

Summary updated

Abstract

Object detection in dark conditions has always been a great challenge due to the complex formation process of low-light images. Currently, the mainstream methods usually adopt domain adaptation with Teacher-Student architecture to solve the dark object detection problem, and they imitate the dark conditions by using non-learnable data augmentation strategies on the annotated source daytime images. Note that these methods neglected to model the intrinsic imaging process, i.e. image signal processing (ISP), which is important for camera sensors to generate low-light images. To solve the above problems, in this paper, we propose a novel method named ISP-Teacher for dark object detection by exploring Teacher-Student architecture from a new perspective (i.e. self-supervised learning based ISP degradation). Specifically, we first design a day-to-night transformation module that consistent with the ISP pipeline of the camera sensors (ISP-DTM) to make the augmented images look more in line with the natural low-light images captured by cameras, and the ISP-related parameters are learned in a self-supervised manner. Moreover, to avoid the conflict between the ISP degradation and detection tasks in a shared encoder, we propose a disentanglement regularization (DR) that minimizes the absolute value of cosine similarity to disentangle two tasks and push two gradients vectors as orthogonal as possible. Extensive experiments conducted on two benchmarks show the effectiveness of our method in dark object detection. In particular, ISP-Teacher achieves an improvement of +2.4% AP and +3.3% AP over the SOTA method on BDD100k and SHIFT datasets, respectively. The code can be found at https://github.com/zhangyin1996/ISP-Teacher.
BibTeX
@inproceedings{zhang2024isp,
  author = {Yin Zhang and Yongqiang Zhang and Zian Zhang and Man Zhang and Rui Tian and Mingli Ding},
  title = {{ISP-Teacher: Image Signal Process with Disentanglement Regularization for Unsupervised Domain Adaptive Dark Object Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=GUnazwPt7m},
}
Figure