Computationally Efficient Target Classification in Multispectral Image Data with Deep Neural Networks

Lukas Cavigelli, Dominic Bernath, Michele Magno, Luca BeniniarXiv 2016 · DOI · OpenReview

Abstract

Detecting and classifying targets in video streams from surveillance cameras is a cumbersome, error-prone and expensive task. Often, the incurred costs are prohibitive for real-time monitoring. This leads to data being stored locally or transmitted to a central storage site for post-incident examination. The required communication links and archiving of the video data are still expensive and this setup excludes preemptive actions to respond to imminent threats. An effective way to overcome these limitations is to build a smart camera that transmits alerts when relevant video sequences are detected. Deep neural networks (DNNs) have come to outperform humans in visual classifications tasks. The concept of DNNs and Convolutional Networks (ConvNets) can easily be extended to make use of higher-dimensional input data such as multispectral data. We explore this opportunity in terms of achievable accuracy and required computational effort. To analyze the precision of DNNs for scene labeling in an urban surveillance scenario we have created a dataset with 8 classes obtained in a field experiment. We combine an RGB camera with a 25-channel VIS-NIR snapshot sensor to assess the potential of multispectral image data for target classification. We evaluate several new DNNs, showing that the spectral information fused together with the RGB frames can be used to improve the accuracy of the system or to achieve similar accuracy with a 3x smaller computation effort. We achieve a very high per-pixel accuracy of 99.1%. Even for scarcely occurring, but particularly interesting classes, such as cars, 75% of the pixels are labeled correctly with errors occurring only around the border of the objects. This high accuracy was obtained with a training set of only 30 labeled images, paving the way for fast adaptation to various application scenarios.

Introduction

Traditional surveillance systems are often "dumb"—they stream massive amounts of raw video to central servers for storage or post-incident analysis. This is expensive, bandwidth-heavy, and prevents real-time preemptive action. The solution is a "smart camera" that processes data on-site. However, running high-accuracy Deep Neural Networks (DNNs) on the limited hardware of an embedded camera is a major challenge. Standard RGB cameras struggle to distinguish between materials that look similar in visible light but have different physical properties. This research investigates whether adding multispectral channels (specifically in the Near-Infrared range) can make classification both more accurate and more computationally efficient for embedded platforms.

Methods and Materials

The researchers developed a hardware-software pipeline to fuse spatial and spectral data. They combined a high-resolution RGB camera with a 25-channel VIS-NIR (600–975 nm) snapshot mosaic sensor.

Hardware setup and multispectral data acquisition showing the RGB and spectral bands.

Because the sensors have different resolutions and viewpoints, a Local Weighted Mean Transform (LWMT) and bicubic interpolation were used to warp the multispectral data onto the RGB frames, creating a 28-channel "image cube." The study evaluated three types of networks: simple per-pixel MLPs, standard ConvNets for shape and texture, and ResNet-inspired architectures to optimize the trade-off between error rate and computational cost.

A custom dataset was created consisting of 40 high-resolution images from a street surveillance perspective, covering 8 categories: car/truck, sky, building, road/gravel, tree/shrubbery, tram, water, and distant background. Labeling was performed using a semi-automated approach with SLIC superpixels to handle the high pixel count efficiently.

Key Results and Discussion

The best-performing ConvNet achieved a 99.1% per-pixel accuracy when using combined RGB+Multispectral data. Adding multispectral data to the best RGB model reduced the error rate from 1.3% to 0.9%, resulting in an 18% improvement in identifying rare but important classes like cars and trucks.

Scene labeling output showing the segmented classification of urban elements.

A significant finding is the "Spectral-Computational Trade-off": multispectral data can compensate for a simpler neural network. A lightweight network using multispectral data can match the performance of a much heavier RGB-only network. Specifically, the "Network C1" achieved the same accuracy as the baseline RGB-only model but required 3x less computation, facilitating real-time processing on mobile processors like the Nvidia Tegra K1.

Conclusion

This study demonstrates that multispectral sensors allow for a 3x reduction in computational burden for the same accuracy level. While RGB data is excellent for identifying shapes, multispectral data is superior for identifying materials, such as distinguishing natural vegetation from green man-made objects. This high accuracy was obtained with a training set of only 30 labeled images, paving the way for fast adaptation to various application scenarios. Future work should expand the dataset to include diverse weather and lighting conditions and explore dynamic environments where camera movement complicates sensor alignment.

Summary updated

Abstract

Detecting and classifying targets in video streams from surveillance cameras is a cumbersome, error-prone and expensive task. Often, the incurred costs are prohibitive for real-time monitoring. This leads to data being stored locally or transmitted to a central storage site for post-incident examination. The required communication links and archiving of the video data are still expensive and this setup excludes preemptive actions to respond to imminent threats. An effective way to overcome these limitations is to build a smart camera that transmits alerts when relevant video sequences are detected. Deep neural networks (DNNs) have come to outperform humans in visual classifications tasks. The concept of DNNs and Convolutional Networks (ConvNets) can easily be extended to make use of higher-dimensional input data such as multispectral data. We explore this opportunity in terms of achievable accuracy and required computational effort. To analyze the precision of DNNs for scene labeling in an urban surveillance scenario we have created a dataset with 8 classes obtained in a field experiment. We combine an RGB camera with a 25-channel VIS-NIR snapshot sensor to assess the potential of multispectral image data for target classification. We evaluate several new DNNs, showing that the spectral information fused together with the RGB frames can be used to improve the accuracy of the system or to achieve similar accuracy with a 3x smaller computation effort. We achieve a very high per-pixel accuracy of 99.1%. Even for scarcely occurring, but particularly interesting classes, such as cars, 75% of the pixels are labeled correctly with errors occurring only around the border of the objects. This high accuracy was obtained with a training set of only 30 labeled images, paving the way for fast adaptation to various application scenarios.
BibTeX
@misc{cavigelli2016computationally,
  author = {Lukas Cavigelli and Dominic Bernath and Michele Magno and Luca Benini},
  title = {{Computationally Efficient Target Classification in Multispectral Image Data with Deep Neural Networks}},
  year = {2016},
  eprint = {1611.03130},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.03130},
}
Figure