YOLO in the Dark - Domain Adaptation Method for Merging Multiple Models

Yukihiro Sasagawa, Hajime Nagahara

Abstract

Computer vision systems typically struggle in extreme low-light environments where short-exposure images are dominated by noise and long-exposure images suffer from motion blur. While specialized models exist for image enhancement and object detection separately, combining them often requires massive annotated datasets of RAW images, which are expensive to produce. This paper introduces "YOLO in the Dark," a domain adaptation method that merges pre-trained expert models—a low-light enhancer and an object detector—using specialized "glue layers." By leveraging knowledge distillation to create a generative model, the authors train these layers to detect objects in RAW images using only standard RGB datasets. The resulting system achieves significant accuracy gains in near-total darkness while reducing computational costs by nearly 60% compared to traditional sequential pipelines.

Introduction

Standard computer vision pipelines are designed for well-lit RGB environments. In low-light scenarios, the signal-to-noise ratio drops significantly, rendering standard detectors like YOLO ineffective. While models such as "Learning-to-See-in-the-Dark" (SID) can reconstruct clear images from RAW sensor data, simply stacking an enhancer and a detector (the "naive" approach) is computationally expensive and often redundant.

The primary bottleneck in developing end-to-end low-light detectors is data: pairing RAW sensor data with object-level annotations is a labor-intensive process. The authors address this by proposing a method to "glue" existing expert models together, allowing a detector trained on standard RGB data (like COCO) to function effectively on the latent features produced by a RAW image encoder.

Methodology: Merging Experts via Glue Layers

The core of the "YOLO in the Dark" architecture is the transition from pixel-space processing to latent-feature-space merging. The process involves three primary components:

  1. Glue Layers: Instead of processing a fully reconstructed image, the authors extract fragments of the SID encoder and the YOLO detector. A "glue layer" is inserted between them to transform the latent features of the SID encoder into the specific format required by the YOLO backbone.
  2. Generative Model ($G2e$): To train the glue layer without new RAW labels, the authors developed a student model designed to be the inverse of the SID decoder. This model takes standard RGB images and generates "fake" latent features that mimic the output of a SID encoder processing a RAW image.
  3. Latent Feature Training: Using the COCO and PASCAL VOC datasets, RGB images are passed through the $G2e$ model to produce synthetic RAW features. These are then used to train the glue layer to map these features to the original YOLO latent space, effectively teaching the model to "see" RAW data using only RGB labels.
Architecture of the glue layer merging two domains.

A critical insight during development was the importance of frequency preservation. The authors found that high-level abstract features alone were insufficient for accurate detection. To maintain the sharp edges and shapes necessary for recognition, they concatenated features from multiple scales (ranging from 1/1 to 1/8) within the glue layer architecture.

Experimental Results

The proposed method was evaluated using the See-in-the-Dark (SID) dataset for RAW low-light testing and standard datasets like COCO for training.

Detection Performance

"YOLO in the Dark" achieved a mean Average Precision (mAP) of 0.55, representing a 2.1x improvement over a naive YOLO model (0.26) applied to enhanced images. The system demonstrated extreme sensitivity, successfully detecting objects in environments with illuminance as low as 0.013 lux. This capability effectively reduces the required exposure time by 60% compared to standard methods.

Object detection results on a bookshelf in low-light conditions.

Computational Efficiency

By bypassing the heavy SID decoder—which is responsible for reconstructing the image for human viewing—the model saves significant processing power. The computational load was reduced from 270.79 GMACs (for a sequential SID + YOLO pipeline) to 113.90 GMACs, making it much more suitable for real-time edge applications.

Conclusion and Future Work

"YOLO in the Dark" demonstrates that merging expert models in the latent feature space is a viable and efficient alternative to end-to-end training on specialized datasets. While the model currently trails slightly behind the absolute accuracy of the full sequential SID+YOLO approach, the massive reduction in computational overhead makes it a compelling solution for low-power vision systems. Future research will focus on refining glue layer optimization to close the accuracy gap and exploring the application of this domain adaptation method to other modalities, such as infrared or thermal imaging.

Summary updated

Abstract

Generating models to handle new visual tasks requires additional datasets, which take considerable effort to create. We propose a method of domain adaptation for merging multiple models with less effort than creating an additional dataset. This method merges pre-trained models in different domains using glue layers and a generative model, which feeds latent features to the glue layers to train them without an additional dataset. We also propose a generative model that is created by distilling knowledge from pre-trained models. This enables the dataset to be reused to create latent features for training the glue layers. We apply this method to object detection in a low-light situation. The YOLO-in-the-Dark model comprises two models, Learning-to-See-in-the-Dark model and YOLO. We present the proposed method and report the result of domain adaptation to detect objects from RAW short-exposure low-light images. The YOLO-in-the-Dark model uses fewer computing resources than the naive approach.
BibTeX
@inproceedings{sasagawa2020yolo,
  author = {Yukihiro Sasagawa and Hajime Nagahara},
  title = {{YOLO in the Dark - Domain Adaptation Method for Merging Multiple Models}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58589-1\_21},
}
Figure