TL;DR
This paper explores how to effectively train deep learning models for video surveillance that use both visible (VIS) and thermal infrared (IR) camera data. They found that augmenting visible-light training data with specific transformations, like inverse luminance and Gaussian blur, significantly improves the model's ability to detect objects in combined VIS/IR imagery, making it more robust for day-and-night surveillance.
What Problem Does This Paper Address?
Intelligent video surveillance systems often rely on cameras that capture images in different spectral ranges, such as visible light (VIS) and thermal infrared (IR). While VIS cameras provide color and texture information, they struggle in low-light conditions. IR cameras, on the other hand, excel in darkness by detecting heat signatures but lack color and can be affected by reflections. Combining these sensors offers a more comprehensive view, but training deep neural networks (CNNs) for multispectral object detection is challenging. Specifically, obtaining large, annotated IR datasets is difficult and expensive. This paper addresses the problem of how to best leverage readily available VIS data to train models that perform well on both VIS and IR imagery, or a combination of both, for surveillance tasks.
Methods
The core of this paper lies in investigating various data augmentation techniques applied to visible light (RGB) images to make them more compatible with thermal infrared (IR) data for training a CNN. Instead of transforming IR data to match VIS, they transform VIS data to resemble IR. They explore several color space transformations, including:
- Luminance: A weighted average of RGB channels, mimicking how the human eye perceives brightness.
- Intensity: The average of RGB channels.
- Value: The maximum value among RGB channels.
These transformations are then combined with other augmentation techniques:
- Gaussian Blur: To introduce blurring, similar to how IR images can appear less sharp.
- Inversion: Flipping the color or intensity values.
- Brightness/Contrast Adjustment: Modifying the overall lightness or the range of pixel values.
The authors then train a Convolutional Neural Network (CNN) on these augmented VIS datasets and evaluate its performance on IR, VIS, and combined IR/VIS test sets. They also analyze the filters learned by the first convolutional layer to understand how different augmentations affect the network's feature extraction.
Datasets
The primary dataset used in this study is the ThermalWorld dataset.
- It consists of 1659 pairs of aligned color (VIS) and thermal infrared (IR) images.
- The images are annotated with ground truth for 9 classes: car, truck, minibus, bus, building, human, cat, dog, and boat.
- Image sizes vary, with VIS images at 1185x889x3 pixels and IR images at 640x480x1 pixels.
- Approximately 21% of the data is used for validation.
Key Insights and Ideas
- VIS data can be effectively adapted for IR tasks: The study demonstrates that by applying specific augmentations to VIS images, they can be made "more looking like infrared ones," leading to improved performance on IR and multispectral detection tasks.
- Inverse Luminance + Gaussian Blur is a winning combination: The most effective augmentation strategy found was transforming RGB images to inverse Luminance and then applying Gaussian blur. This suggests that mimicking the intensity distribution and some level of blur present in IR images is crucial.
- Color is less important than intensity/structure for IR adaptation: While VIS images have color, the augmentations that focused on intensity transformations (like Luminance) and structural changes (like blur) yielded better results for IR-related tasks than simply using the raw RGB data.
- Augmentation can bridge the gap between VIS and IR: The research highlights that augmentation is a powerful tool to overcome the limitations of scarce IR data by leveraging abundant VIS data, thereby enabling more robust multispectral surveillance systems.
- Filter visualization provides interpretability: Analyzing the learned filters in the CNN's first convolutional layer showed a growing similarity to IR filters as more effective augmentations were applied to the VIS training data, offering insight into why these augmentations work.
Key Results
- The best performing augmentation technique (inverse Luminance with Gaussian blur) achieved an improvement of approximately 24% in classification accuracy for combined IR/VIS test data compared to training with unaugmented VIS data.
- Compared to training with IR data alone, the augmented VIS approach improved accuracy by about 19% on combined IR/VIS test data.
- For IR-only test data, the best augmentation strategy resulted in a 24% deterioration compared to training with IR data, but this is still a significant improvement over using unaugmented VIS data, which showed a 76% deterioration. This highlights the trade-off: while not perfectly matching IR-trained performance, it allows for using readily available VIS data.
- Raw RGB training data performed surprisingly well on VIS test data (99% accuracy) but poorly on IR test data (33% accuracy).
- Augmentations like adjusting brightness or contrast were found to be less effective or even detrimental compared to intensity and blur transformations.
Limitations and Future Work
- Limitations:
- While the augmented VIS approach significantly improved performance on IR and IR/VIS data, it did not fully match the performance of training directly with IR data for IR-only test sets.
- The paper notes that adapting brightness or contrast was contrary to expectations not helpful.
- Future Work:
- The authors suggest exploring other color systems and methods like gamma correction for further improvements.
- Investigating more advanced augmentation techniques could lead to even better performance.
- The study focused on classification; extending this to object detection tasks would be a natural next step.
Summary updated