Augmentation techniques for video surveillance in the visible and thermal spectral range

Vanessa Buhrmester, Ann-Kristin Grosselfinger, David Munch, Michael Arens

TL;DR

Training deep learning models for thermal infrared (IR) surveillance is heavily bottlenecked by a lack of annotated IR datasets. This paper demonstrates that by applying simple, deterministic augmentations—specifically grayscale conversion, pixel inversion, and Gaussian blur—to widely available visible-spectrum (VIS) data, we can train networks that generalize remarkably well to both thermal and mixed multispectral test environments.

What Problem Does This Paper Address?

Robust 24/7 video surveillance relies on combining visible-light (VIS) cameras with long-wave infrared (IR) sensors to handle challenging conditions like total darkness, deep shadows, and weather occlusions. While visible light captures rich color and texture, thermal sensors capture heat signatures, making them ideal for night monitoring.

However, training Convolutional Neural Networks (CNNs) for object detection requires massive amounts of labeled data. While annotated VIS datasets are plentiful, compatible annotated IR datasets are rare and tedious to produce. This paper tackles a practical question: Can we train a network using easily obtainable visible-spectrum data and use data augmentation to make it perform well on thermal infrared images?

Methods

Rather than using complex generative architectures (like GANs) to translate images between domains, the authors investigate lightweight, deterministic image manipulation techniques. They transform VIS images to mimic the visual properties of IR images using the following pipeline:

  1. Color Space Conversions: Transforming RGB images into grayscale using three different methods: Luminance (weighted by human eye sensitivity), Intensity (simple average of channels), and Value (maximum channel value).
  2. Blurring: Applying Gaussian blur ($\sigma = 0.5$) to simulate the softer textures and lack of fine detail typical of thermal sensors.
  3. Inversion: Inverting pixel values to mimic how hot objects (like humans or car engines) often appear bright against cooler backgrounds in thermal imagery.
  4. Brightness & Contrast Adjustments: Tweaking image dynamics to match sensor variations.

The authors train a standard CNN architecture (2 convolutional layers, 2 max-pooling layers, and 3 fully connected layers) on these varied augmentations to isolate exactly how each modification impacts classification accuracy.

Datasets

The primary benchmark used for training and evaluation is the ThermalWorld dataset.

  • It consists of 1,659 pairs of aligned color (VIS) and thermal (LWIR) images.
  • It features 9 classes: car, truck, minibus, bus, building, human, cat, dog, and boat.
  • The authors also reference examples from the MIL Tokyo, DGB Toronto, and CATS datasets to highlight the real-world edge cases where VIS and IR sensors complement each other (e.g., VIS seeing through glass, IR seeing through smoke/darkness).

Key Insights and Ideas

  • Bridging the Texture Gap: Thermal images inherently lack sharp edges and fine textures. The authors found that adding Gaussian blur to VIS training data slightly hurts VIS test performance but significantly boosts IR test performance because it forces the network to learn shape rather than high-frequency texture.
  • The Power of Inversion: Simply converting an image to grayscale isn't enough. Inverting the grayscale image provides a massive performance boost because it aligns the intensity profiles of hot objects with their bright VIS counterparts.
  • Filter Alignment (The "Aha!" Moment): By visualizing the 64 filter kernels of the CNN's first convolutional layer, the authors discovered that as more augmentations (Luminance + Inversion + Blur) are applied to the VIS training data, the learned filters naturally evolve to look almost identical to filters trained directly on raw IR data.

Key Results

The quantitative results highlight the dramatic impact of the right augmentation combination (Luminance + Inversion + Blur):

  • Mixed IR/VIS Testing: When evaluating on a 50/50 mix of visible and thermal images, training with the optimal augmentation combo achieved 78.5% accuracy, outperforming unaugmented RGB training (66%) and even pure IR training (63.5%). This represents a ~19% relative improvement over standard VIS training.
  • Pure IR Testing: If tested strictly on thermal images, a model trained on raw RGB data achieves a poor 33% accuracy. However, training on the augmented VIS data boosts this to 58% accuracy—a massive 76% relative improvement without ever showing the network a real thermal image during training.

Limitations and Future Work

While the proposed method significantly closes the domain gap, a model trained on augmented VIS data (58% accuracy on IR) still lags behind a model trained directly on pure IR data (76% accuracy).

Furthermore, some intuitive augmentations failed; adjusting brightness and contrast actually had a negative effect on accuracy, contrary to the authors' expectations. For future work, the authors suggest exploring more sophisticated color mapping techniques, such as gamma correction or alternative color spaces, to further refine the VIS-to-IR translation.

Summary updated

Abstract

In intelligent video surveillance, cameras record image sequences during day and night. Commonly, this demands different sensors. To achieve a better performance it is not unusual to combine them. We focus on the case that a long-wave infrared camera records continuously and in addition to this, another camera records in the visible spectral range during daytime and an intelligent algorithm supervises the picked up imagery. More accurate, our task is multispectral CNN-based object detection. At first glance, images originating from the visible spectral range differ between thermal infrared ones in the presence of color and distinct texture information on the one hand and in not containing information about thermal radiation that emits from objects on the other hand. Although color can provide valuable information for classification tasks, effects such as varying illumination and specialties of different sensors still represent significant problems. Anyway, obtaining sufficient and practical thermal infrared datasets for training a deep neural network poses still a challenge. That is the reason why training with the help of data from the visible spectral range could be advantageous, particularly if the data, which has to be evaluated contains both visible and infrared data. However, there is no clear evidence of how strongly variations in thermal radiation, shape, or color information influence classification accuracy. To gain deeper insight into how Convolutional Neural Networks make decisions and what they learn from different sensor input data, we investigate the suitability and robustness of different augmentation techniques...
BibTeX
@misc{buhrmester2026augmentation,
  author = {Vanessa Buhrmester and Ann-Kristin Grosselfinger and David Munch and Michael Arens},
  title = {{Augmentation techniques for video surveillance in the visible and thermal spectral range}},
  year = {2026},
  eprint = {2606.13042},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.13042},
}

Loading related content...

Figure