TL;DR
Training deep learning models for thermal infrared (IR) surveillance is heavily bottlenecked by a lack of annotated IR datasets. This paper demonstrates that by applying simple, deterministic augmentations—specifically grayscale conversion, pixel inversion, and Gaussian blur—to widely available visible-spectrum (VIS) data, we can train networks that generalize remarkably well to both thermal and mixed multispectral test environments.
What Problem Does This Paper Address?
Robust 24/7 video surveillance relies on combining visible-light (VIS) cameras with long-wave infrared (IR) sensors to handle challenging conditions like total darkness, deep shadows, and weather occlusions. While visible light captures rich color and texture, thermal sensors capture heat signatures, making them ideal for night monitoring.
However, training Convolutional Neural Networks (CNNs) for object detection requires massive amounts of labeled data. While annotated VIS datasets are plentiful, compatible annotated IR datasets are rare and tedious to produce. This paper tackles a practical question: Can we train a network using easily obtainable visible-spectrum data and use data augmentation to make it perform well on thermal infrared images?
Methods
Rather than using complex generative architectures (like GANs) to translate images between domains, the authors investigate lightweight, deterministic image manipulation techniques. They transform VIS images to mimic the visual properties of IR images using the following pipeline:
- Color Space Conversions: Transforming RGB images into grayscale using three different methods: Luminance (weighted by human eye sensitivity), Intensity (simple average of channels), and Value (maximum channel value).
- Blurring: Applying Gaussian blur ($\sigma = 0.5$) to simulate the softer textures and lack of fine detail typical of thermal sensors.
- Inversion: Inverting pixel values to mimic how hot objects (like humans or car engines) often appear bright against cooler backgrounds in thermal imagery.
- Brightness & Contrast Adjustments: Tweaking image dynamics to match sensor variations.
The authors train a standard CNN architecture (2 convolutional layers, 2 max-pooling layers, and 3 fully connected layers) on these varied augmentations to isolate exactly how each modification impacts classification accuracy.
Datasets
The primary benchmark used for training and evaluation is the ThermalWorld dataset.
- It consists of 1,659 pairs of aligned color (VIS) and thermal (LWIR) images.
- It features 9 classes: car, truck, minibus, bus, building, human, cat, dog, and boat.
- The authors also reference examples from the MIL Tokyo, DGB Toronto, and CATS datasets to highlight the real-world edge cases where VIS and IR sensors complement each other (e.g., VIS seeing through glass, IR seeing through smoke/darkness).
Key Insights and Ideas
- Bridging the Texture Gap: Thermal images inherently lack sharp edges and fine textures. The authors found that adding Gaussian blur to VIS training data slightly hurts VIS test performance but significantly boosts IR test performance because it forces the network to learn shape rather than high-frequency texture.
- The Power of Inversion: Simply converting an image to grayscale isn't enough. Inverting the grayscale image provides a massive performance boost because it aligns the intensity profiles of hot objects with their bright VIS counterparts.
- Filter Alignment (The "Aha!" Moment): By visualizing the 64 filter kernels of the CNN's first convolutional layer, the authors discovered that as more augmentations (Luminance + Inversion + Blur) are applied to the VIS training data, the learned filters naturally evolve to look almost identical to filters trained directly on raw IR data.
Key Results
The quantitative results highlight the dramatic impact of the right augmentation combination (Luminance + Inversion + Blur):
- Mixed IR/VIS Testing: When evaluating on a 50/50 mix of visible and thermal images, training with the optimal augmentation combo achieved 78.5% accuracy, outperforming unaugmented RGB training (66%) and even pure IR training (63.5%). This represents a ~19% relative improvement over standard VIS training.
- Pure IR Testing: If tested strictly on thermal images, a model trained on raw RGB data achieves a poor 33% accuracy. However, training on the augmented VIS data boosts this to 58% accuracy—a massive 76% relative improvement without ever showing the network a real thermal image during training.
Limitations and Future Work
While the proposed method significantly closes the domain gap, a model trained on augmented VIS data (58% accuracy on IR) still lags behind a model trained directly on pure IR data (76% accuracy).
Furthermore, some intuitive augmentations failed; adjusting brightness and contrast actually had a negative effect on accuracy, contrary to the authors' expectations. For future work, the authors suggest exploring more sophisticated color mapping techniques, such as gamma correction or alternative color spaces, to further refine the VIS-to-IR translation.
Summary updated