Partially fake it till you make it: mixing real and fake thermal images for improved object detection

Francesco Bongini, Lorenzo Berlincioni, Marco Bertini, Alberto Del Bimbo

Introduction

Object detection serves as the primary sensory mechanism for autonomous vehicles. However, conventional RGB cameras frequently fail under adverse conditions such as total darkness, heavy fog, or intense glare. Thermal (long-wave infrared) cameras offer a robust alternative, as they rely on heat signatures rather than visible light. Despite their advantages, thermal imaging faces a significant "data bottleneck." While visible-light datasets like COCO or ImageNet contain millions of annotated images, thermal datasets are often orders of magnitude smaller, hindering the training of high-performance deep learning models.

Comparison of RGB and thermal imaging in challenging lighting conditions. Thermal sensors provide clear visibility where RGB cameras suffer from glare and low light.

The obvious solution—simulation—is notoriously difficult in the thermal domain. Unlike RGB simulation, which primarily models light transport, thermal simulation requires complex thermodynamic modeling of materials, engine heat dissipation, and environmental temperature fluctuations. Consequently, creating hyper-realistic synthetic thermal environments is computationally prohibitive.

Proposed Methodology

To address the data scarcity without the overhead of full-scale thermal simulation, this research proposes a hybrid "partial" simulation pipeline. The core idea is to combine real-world thermal backgrounds with synthetic foreground objects.

Data Compositing and GAN Refinement

The authors utilized real thermal images from the FLIR ADAS dataset as backgrounds. Using the Unity game engine, they "dropped in" 3D models of pedestrians and vehicles into these scenes. This approach provides perfect ground-truth labels automatically. However, raw 3D renders often lack the nuanced textures of real thermal imagery, creating a domain gap that can confuse neural networks.

To bridge this gap, a CycleGAN-based architecture was employed. The GAN translates synthetic objects into the specific "style" of real thermal data. Interestingly, the GAN learned to incorporate physical thermal properties not present in the original 3D models, such as the thermal reflection of a vehicle's engine heat on the pavement.

The proposed data augmentation and synthesis pipeline, showing the integration of bounding box augmentation, random augmentation, and GAN-based synthetic data generation.

Augmentation Strategies

The study also evaluated the efficacy of standard RGB data augmentation techniques in the thermal domain:

  • RandAugment: Randomly applying a sequence of image filters.
  • BBox Augment (BbAug): Applying transformations specifically within the object bounding boxes.
  • Thermal Adaptation: A layer-wise adaptation of the YOLOv3 architecture was used as the baseline detector to evaluate these strategies.

Experimental Results

The hybrid approach demonstrated significant performance gains on the FLIR ADAS benchmark. The baseline model, trained only on real data, achieved a mean Average Precision (mAP) of 73.2%. Incorporating standard bounding box augmentation improved this to 75.0%. The proposed method, utilizing GAN-refined synthetic data, reached a state-of-the-art 76.4% mAP.

Notably, this thermal-only model outperformed several multispectral systems that utilize both thermal and visible-light sensors. This suggests that high-quality, targeted data for a single sensor can be more effective than simply adding more sensors with suboptimal data.

Key Insights

  1. The 20% Rule: The researchers identified a "sweet spot" for data mixing. Adding approximately 10-20% synthetic data provided the optimal performance boost. Exceeding this threshold (e.g., 50% synthetic) led to performance degradation, likely due to the model overfitting to simulator-specific artifacts.
  2. Domain-Specific Augmentation: Not all RGB techniques translate well. For example, color inversion (solarization) can be detrimental to thermal detectors, which rely on specific intensity gradients that represent heat.
  3. Emergent Physical Realism: The GAN's ability to "learn" thermal reflections indicates that generative models can capture complex physical interactions that are difficult to program manually in a simulator.

Conclusion and Future Work

This research demonstrates that "partially faking" data through smart compositing and GAN refinement is a viable and efficient strategy for overcoming data scarcity in specialized domains. While the current results are promising, future work will focus on expanding the diversity of 3D assets and developing "domain-aware" GANs capable of simulating varied weather conditions, such as the effect of rain on thermal signatures. This methodology holds potential for other data-starved fields, including medical imaging and satellite-based surveillance.

Summary updated

Abstract

In this paper we propose a novel data augmentation approach for visual content domains that have scarce training datasets, compositing synthetic 3D objects within real scenes. We show the performance of the proposed system in the context of object detection in thermal videos, a domain where 1) training datasets are very limited compared to visible spectrum datasets and 2) creating full realistic synthetic scenes is extremely cumbersome and expensive due to the difficulty in modeling the thermal properties of the materials of the scene. We compare different augmentation strategies, including state of the art approaches obtained through RL techniques, the injection of simulated data and the employment of a generative model, and study how to best combine our proposed augmentation with these other techniques.Experimental results demonstrate the effectiveness of our approach, and our single-modality detector achieves state-of-the-art results on the FLIR ADAS dataset.
BibTeX
@inproceedings{bongini2021partially,
  author = {Francesco Bongini and Lorenzo Berlincioni and Marco Bertini and Alberto Del Bimbo},
  title = {{Partially fake it till you make it: mixing real and fake thermal images for improved object detection}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=5OV3wzft4O},
}
Figure