Fast and Easy Sensor Adaptation With Self-Training

Jinhyuk Choi, Byeongju Lee, Seho Shin, Daehyun JiIEEE Access 2023 · OpenReview

Abstract

Object detectors based on deep neural networks have the disadvantage that new labels should be acquired whenever the complementary metal-oxide semiconductor (CMOS) image sensor (CIS) is changed. In this study, we propose a fast and easy two-step sensor-adaptation method without labels for the target domain; 1) simple adaptation, and 2) self-training. The simple-adaptation process transfers the knowledge of the source model to the target model by updating the batch normalization parameters, and matches the feature distributions of the source domain and those of target domain. In the self-training process, we employ the ensemble model strategy to mitigate the over-fitting problem using noisy pseudo labels generated by the simple-adaptation model. Quantitative and qualitative experiments show that the proposed method can transfer the knowledge from one CIS model to another, even if the data format of the target domain is different from that of the source CIS domain.

Introduction

Deep learning-based object detectors are highly sensitive to the hardware used for image capture. When a manufacturer updates a CMOS image sensor (CIS) or modifies Image Signal Processor (ISP) settings, the resulting visual characteristics—such as color, contrast, and brightness—shift. This domain shift typically leads to a significant drop in detection accuracy. Traditionally, adapting to a new sensor required the labor-intensive collection and manual labeling of thousands of images. This paper introduces a two-step, unsupervised method to adapt models to new sensors or data formats (like raw Bayer patterns) without requiring any new manual labels.

Methodology

The proposed framework consists of two primary stages: Simple Adaptation and Ensemble Self-Training.

  1. Simple Adaptation: This stage serves as an initial alignment. Instead of full retraining, only the Batch Normalization (BN) parameters are updated using unlabeled data from the target sensor. Since BN layers capture the mean and variance of features, recalibrating them aligns the target domain's feature distribution with the source model's expectations.
  2. Ensemble Self-Training: To refine performance, the model generates pseudo-labels for the target data. To mitigate the impact of noisy labels (false positives or misses), an ensemble strategy is employed. Three different confidence thresholds are used to create three distinct training sets: high-precision/low-recall, moderate, and low-precision/high-recall. Three sub-models are trained on these sets and then merged into a single ensemble model via a weighted sum of parameters. This process can be iterated for further refinement.
Overview of the proposed two-step sensor adaptation framework.

Experimental Results

The method was evaluated on several datasets, including a private dataset of over 100,000 images (Samsung S5K2G1 sensor in raw and RGB formats), the Cityscapes dataset (converted to pseudo-raw), and specialized low-brightness scenes.

On the private raw dataset, the source model (trained on different RGB data) initially failed completely (AP of 0.000). Simple Adaptation immediately boosted the AP to 0.935, and two iterations of self-training further increased it to 0.953. On the Cityscapes 8-bit raw benchmark, the proposed method achieved an AP of 0.696, significantly outperforming existing domain adaptation methods like GPA (0.110) and AT (0.608).

Qualitative detection results in challenging highway and tunnel environments.

Qualitative analysis shows the model's robustness in challenging environments, such as tunnels and night scenes, where it successfully detects vehicles that the baseline model misses due to low contrast.

Conclusion

This research demonstrates that Batch Normalization recalibration and ensemble-based self-training provide a powerful, label-free solution for sensor adaptation. The ability to adapt models directly to raw sensor data is particularly valuable, as it allows for bypassing the latency and power costs of traditional ISP pipelines. Future work will focus on expanding the method to more diverse object classes and optimizing the self-training loop for real-time edge deployment.

Summary updated

Abstract

Object detectors based on deep neural networks have the disadvantage that new labels should be acquired whenever the complementary metal-oxide semiconductor (CMOS) image sensor (CIS) is changed. In this study, we propose a fast and easy two-step sensor-adaptation method without labels for the target domain; 1) simple adaptation, and 2) self-training. The simple-adaptation process transfers the knowledge of the source model to the target model by updating the batch normalization parameters, and matches the feature distributions of the source domain and those of target domain. In the self-training process, we employ the ensemble model strategy to mitigate the over-fitting problem using noisy pseudo labels generated by the simple-adaptation model. Quantitative and qualitative experiments show that the proposed method can transfer the knowledge from one CIS model to another, even if the data format of the target domain is different from that of the source CIS domain.
BibTeX
@inproceedings{choi2023fast,
  author = {Jinhyuk Choi and Byeongju Lee and Seho Shin and Daehyun Ji},
  title = {{Fast and Easy Sensor Adaptation With Self-Training}},
  booktitle = {IEEE Access},
  year = {2023},
  url = {https://openreview.net/forum?id=9lZS4t3XRf},
}
Figure