Augmentation techniques for video surveillance in the visible and thermal spectral range

Vanessa Buhrmester, Ann-Kristin Grosselfinger, David Münch, Michael Arens

TL;DR

This paper explores how to effectively train deep learning models for video surveillance that use both visible (VIS) and thermal infrared (IR) camera data. They found that augmenting visible-light training data with specific transformations, like inverse luminance and Gaussian blur, significantly improves the model's ability to detect objects in combined VIS/IR imagery, making it more robust for day-and-night surveillance.

What Problem Does This Paper Address?

Intelligent video surveillance systems often rely on cameras that capture images in different spectral ranges, such as visible light (VIS) and thermal infrared (IR). While VIS cameras provide color and texture information, they struggle in low-light conditions. IR cameras, on the other hand, excel in darkness by detecting heat signatures but lack color and can be affected by reflections. Combining these sensors offers a more comprehensive view, but training deep neural networks (CNNs) for multispectral object detection is challenging. Specifically, obtaining large, annotated IR datasets is difficult and expensive. This paper addresses the problem of how to best leverage readily available VIS data to train models that perform well on both VIS and IR imagery, or a combination of both, for surveillance tasks.

Methods

The core of this paper lies in investigating various data augmentation techniques applied to visible light (RGB) images to make them more compatible with thermal infrared (IR) data for training a CNN. Instead of transforming IR data to match VIS, they transform VIS data to resemble IR. They explore several color space transformations, including:

  • Luminance: A weighted average of RGB channels, mimicking how the human eye perceives brightness.
  • Intensity: The average of RGB channels.
  • Value: The maximum value among RGB channels.

These transformations are then combined with other augmentation techniques:

  • Gaussian Blur: To introduce blurring, similar to how IR images can appear less sharp.
  • Inversion: Flipping the color or intensity values.
  • Brightness/Contrast Adjustment: Modifying the overall lightness or the range of pixel values.

The authors then train a Convolutional Neural Network (CNN) on these augmented VIS datasets and evaluate its performance on IR, VIS, and combined IR/VIS test sets. They also analyze the filters learned by the first convolutional layer to understand how different augmentations affect the network's feature extraction.

Datasets

The primary dataset used in this study is the ThermalWorld dataset.

  • It consists of 1659 pairs of aligned color (VIS) and thermal infrared (IR) images.
  • The images are annotated with ground truth for 9 classes: car, truck, minibus, bus, building, human, cat, dog, and boat.
  • Image sizes vary, with VIS images at 1185x889x3 pixels and IR images at 640x480x1 pixels.
  • Approximately 21% of the data is used for validation.

Key Insights and Ideas

  • VIS data can be effectively adapted for IR tasks: The study demonstrates that by applying specific augmentations to VIS images, they can be made "more looking like infrared ones," leading to improved performance on IR and multispectral detection tasks.
  • Inverse Luminance + Gaussian Blur is a winning combination: The most effective augmentation strategy found was transforming RGB images to inverse Luminance and then applying Gaussian blur. This suggests that mimicking the intensity distribution and some level of blur present in IR images is crucial.
  • Color is less important than intensity/structure for IR adaptation: While VIS images have color, the augmentations that focused on intensity transformations (like Luminance) and structural changes (like blur) yielded better results for IR-related tasks than simply using the raw RGB data.
  • Augmentation can bridge the gap between VIS and IR: The research highlights that augmentation is a powerful tool to overcome the limitations of scarce IR data by leveraging abundant VIS data, thereby enabling more robust multispectral surveillance systems.
  • Filter visualization provides interpretability: Analyzing the learned filters in the CNN's first convolutional layer showed a growing similarity to IR filters as more effective augmentations were applied to the VIS training data, offering insight into why these augmentations work.

Key Results

  • The best performing augmentation technique (inverse Luminance with Gaussian blur) achieved an improvement of approximately 24% in classification accuracy for combined IR/VIS test data compared to training with unaugmented VIS data.
  • Compared to training with IR data alone, the augmented VIS approach improved accuracy by about 19% on combined IR/VIS test data.
  • For IR-only test data, the best augmentation strategy resulted in a 24% deterioration compared to training with IR data, but this is still a significant improvement over using unaugmented VIS data, which showed a 76% deterioration. This highlights the trade-off: while not perfectly matching IR-trained performance, it allows for using readily available VIS data.
  • Raw RGB training data performed surprisingly well on VIS test data (99% accuracy) but poorly on IR test data (33% accuracy).
  • Augmentations like adjusting brightness or contrast were found to be less effective or even detrimental compared to intensity and blur transformations.

Limitations and Future Work

  • Limitations:
    • While the augmented VIS approach significantly improved performance on IR and IR/VIS data, it did not fully match the performance of training directly with IR data for IR-only test sets.
  • The paper notes that adapting brightness or contrast was contrary to expectations not helpful.
  • Future Work:
    • The authors suggest exploring other color systems and methods like gamma correction for further improvements.
  • Investigating more advanced augmentation techniques could lead to even better performance.
  • The study focused on classification; extending this to object detection tasks would be a natural next step.

Summary updated

Abstract

In intelligent video surveillance, cameras record image sequences during day and night. Commonly, this demands different sensors. To achieve a better performance it is not unusual to combine them. We focus on the case that a long-wave infrared camera records continuously and in addition to this, another camera records in the visible spectral range during daytime and an intelligent algorithm supervises the picked up imagery. More accurate, our task is multispectral CNN-based object detection. At first glance, images originating from the visible spectral range differ between thermal infrared ones in the presence of color and distinct texture information on the one hand and in not containing information about thermal radiation that emits from objects on the other hand. Although color can provide valuable information for classification tasks, effects such as varying illumination and specialties of different sensors still represent significant problems. Anyway, obtaining sufficient and practical thermal infrared datasets for training a deep neural network poses still a challenge. That is the reason why training with the help of data from the visible spectral range could be advantageous, particularly if the data, which has to be evaluated contains both visible and infrared data. However, there is no clear evidence of how strongly variations in thermal radiation, shape, or color information influence classification accuracy. To gain deeper insight into how Convolutional Neural Networks make decisions and what they learn from different sensor input data, we investigate the suitability and robustness of different augmentation techniques. We use the publicly available large-scale multispectral ThermalWorld dataset consisting of images in the long-wave infrared and visible spectral range showing persons, vehicles, buildings, and pets and train for image classification a Convolutional Neural Network. The training data will be augmented with several modifications based on their different properties to find out which ones cause which impact and lead to the best classification performance.
BibTeX
@misc{buhrmester2026augmentation,
  author = {Vanessa Buhrmester and Ann-Kristin Grosselfinger and David Münch and Michael Arens},
  title = {{Augmentation techniques for video surveillance in the visible and thermal spectral range}},
  year = {2026},
  eprint = {2606.13042},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.13042},
}
Figure