TL;DR
This paper introduces a novel approach for pedestrian detection in thermal infrared (IR) images, addressing challenges like low resolution and noise. They propose using Local Steering Kernels (LSK) to create "tensor" descriptors and a new "Linear Support Tensor Machine" with a Matrix Cosine Similarity kernel for efficient training and detection. This method achieves fast and accurate pedestrian localization, outperforming existing methods on thermal datasets.
What Problem Does This Paper Address?
Detecting pedestrians in thermal IR images is significantly harder than in visible light images. Thermal images suffer from low resolution, high noise, and suppressed textures, making it difficult to extract reliable features. Traditional methods like Histogram of Oriented Gradients (HOG) struggle with this noisy and less descriptive data. Existing object detection frameworks often require extensive training data and computational resources, which are also challenges in the IR domain. The paper aims to develop a robust and efficient pedestrian detector specifically for thermal IR imagery.
Methods
The core of the proposed method lies in three key contributions:
- Local Steering Kernel (LSK) as a Tensor Descriptor: Instead of traditional HOG, which aggregates orientation statistics, LSK captures local image geometry more directly. It uses a "steering matrix" derived from local gradients to identify principal directions. Crucially, these LSK descriptors are organized into a third-order tensor, allowing for richer representation of local structure. This tensor representation is then decorrelated using Principal Component Analysis (PCA) to reduce dimensionality.
- Linear Support Tensor Machine (LSTM) with Matrix Cosine Similarity (MCS) Kernel: The authors extend the Support Vector Machine (SVM) framework to handle tensor data. They introduce a novel kernel based on Matrix Cosine Similarity (MCS), which is adapted for tensor inputs. This MCS kernel, combined with a maximum margin learning approach, allows for efficient training of a rigid detector template. The training process is described as relatively short and simple compared to other methods.
- Fast Detection via Fourier Transform: To overcome the computational cost of sliding window detection, the paper leverages the tensor representation and the MCS kernel. By transforming the problem into the frequency domain using Fast Fourier Transforms (FFT), they achieve an exact acceleration of the detection process. This allows for very fast and efficient pedestrian localization, even at multiple scales.
Datasets
The proposed method was evaluated on the following datasets:
- OSU Thermal (OSU-T): A dataset of 284 images from 10 sequences.
- OSU Color Thermal (OSU-CT): A dataset of 17,088 thermal and color images from 6 video sequences. The authors created ground truth annotations for this dataset.
- LSI Far Infrared Pedestrian Database: A dataset with both classification and detection setups. The authors used the detection subset.
Key Insights and Ideas
- Tensor Representation for Local Geometry: Organizing LSK descriptors into tensors allows for a more comprehensive capture of local image structure, which is crucial for dealing with the inherent noise and low resolution of thermal images.
- LSK vs. HOG in Thermal Imagery: LSK's ability to capture dominant local orientation directly, rather than relying on aggregated statistics, makes it more robust to noise and texture suppression in thermal images compared to HOG.
- MCS Kernel for Tensor Similarity: Adapting the Matrix Cosine Similarity to tensors provides a powerful and efficient way to measure similarity between detector templates and image patches, leading to a robust learning framework.
- Exact Acceleration via FFT: Transforming the detection process into the frequency domain using FFTs allows for a significant speedup without approximations, making real-time pedestrian detection feasible.
- Rigid Detectors Can Be Effective: The paper demonstrates that a well-designed rigid detector, even without explicit part modeling, can perform very well, especially when dealing with small objects like distant pedestrians.
Key Results
- The proposed LSK-M3CS detector significantly outperforms HOG-based linear SVM detectors on the OSU-T and OSU-CT datasets, achieving lower miss rates.
- On the LSI dataset, the LSK-M3CS detector also shows superior performance compared to HOG root filter and Latent-SVM with parts.
- The method achieves very fast detection times, with a typical detection in a 240x360 image taking about a second, significantly faster than traditional sliding window approaches.
- The paper includes the release of in-house annotations for over 17,000 frames of the OSU Color Thermal database to aid the research community.
Limitations and Future Work
- Partial Occlusion: The paper acknowledges that the detector faces difficulties in detecting partially occluded people, especially in the absence of tracking information or a background model.
- Annotation Challenges: The authors note that the annotation task for thermal datasets can be challenging, particularly with occlusions and varying pedestrian sizes.
- Future Work: While not explicitly detailed, the paper suggests that the proposed framework is general enough to incorporate tracking information and background models for building more sophisticated systems. The exploration of multilinear support tensor machines with higher-order tensor decomposition is also mentioned as a potential avenue, though its justification over the simpler linear model is questioned.
Summary updated