TL;DR
This paper investigates the effectiveness of using far-infrared (FIR) cameras, alone and in combination with visible light cameras, for pedestrian detection across day and night conditions. They find that FIR cameras significantly outperform visible light cameras at night and are competitive even during the day, suggesting FIR is a crucial modality for robust pedestrian detection systems.
What Problem Does This Paper Address?
Pedestrian detection is a critical component for advanced driver-assistance systems (ADAS) and autonomous vehicles. However, its performance is severely hampered by varying lighting conditions, especially at night or in low-visibility scenarios. While visible light cameras are standard, their limitations in poor lighting are well-known. Recent research has explored using non-visible imaging modalities, particularly infrared (IR), to overcome these challenges. This paper specifically addresses the question of how far-infrared (FIR) cameras, which capture thermal radiation, perform for pedestrian detection compared to visible light cameras, and whether combining both modalities offers a significant advantage, particularly across different times of day and night.
Methods
The authors evaluate established pedestrian detection models and features. They focus on three types of models:
- Holistic: A linear Support Vector Machine (SVM) classifier.
- Patch-based: A Random Forest of local experts (RF).
- Part-based: The Deformable Part Model (DPM).
For features, they consider:
- Histograms of Oriented Gradients (HOG).
- Local Binary Patterns (LBP).
- A combination of HOG and LBP (HOG+LBP).
They compare detection performance using:
- Visible light images only.
- FIR images only.
- A combination of visible and FIR images.
For the combined modality approach, they employ an "early fusion" strategy, which involves concatenating feature vectors extracted from both visible and FIR images before feeding them to the classifier.
Datasets
The paper utilizes two datasets:
- CVC-14 Dataset: A new dataset introduced by the authors, consisting of synchronized visible and FIR video sequences captured during both day and night. This dataset includes annotated pedestrian bounding boxes.
- KAIST Multispectral Pedestrian Dataset: A publicly available dataset containing 95K frame pairs from synchronized onboard color and thermal cameras.
Key Insights and Ideas
- FIR Dominance at Night: The most striking finding is that FIR cameras provide significantly better pedestrian detection results at night compared to visible light cameras. This is attributed to the fact that human bodies emit thermal radiation, making them detectable regardless of ambient light.
- Daytime FIR Competitiveness: Surprisingly, FIR cameras also perform competitively with visible light cameras during the daytime. In some cases, FIR even slightly outperforms visible light, suggesting its utility extends beyond nighttime scenarios.
- Modality Fusion Benefits (Daytime): Combining visible and FIR features yields the best detection performance during the daytime, showing a notable improvement over using either modality alone. This suggests that while FIR is strong, visible light still provides complementary information.
- Limited Fusion Gain at Night: At nighttime, FIR alone is already very effective. Combining it with visible light offers only a marginal improvement, indicating that the thermal information is largely sufficient for robust detection in darkness.
- Cost-Effectiveness Consideration: The authors suggest that for systems solely focused on pedestrian detection, improving FIR camera technology might be more cost-effective than relying on complex fusion systems, especially if synchronization and alignment are challenging.
Key Results
- CVC-14 Dataset:
- At night, FIR consistently outperforms visible light across all tested detectors and features. For example, with the RF/HOG+LBP detector, the average miss rate (AMR) is around 24.8% for FIR versus 81.2% for visible light.
- During the day, FIR achieves comparable or better results than visible light. For RF/HOG+LBP, AMR is 16.7% for FIR vs. 26.6% for visible light.
- Combining visible and FIR features provides the best daytime performance (e.g., 16.7% AMR for RF/HOG+LBP), while at night, FIR alone is nearly as good as the combined approach (24.8% vs. 24.8% AMR for RF/HOG+LBP).
- KAIST Dataset:
- Similar trends are observed. For daytime detection of reasonable pedestrians using RF/HOG+LBP, the AMR is 31.5% for FIR, 39.7% for visible, and 28.7% for the combined visible+FIR approach.
- At night, FIR is again superior, with an AMR of 25.3% compared to 76.0% for visible light. The combined approach offers a slight improvement (29.4% AMR).
Limitations and Future Work
- Synchronization and Alignment: While the KAIST dataset offers synchronized and spatially aligned data, the CVC-14 dataset has slight temporal misalignments and camera position differences. The authors argue these are minor for their comparison purposes but acknowledge that precise alignment is crucial for true multi-modal fusion.
- Dataset Specificity: The performance of detectors can be influenced by the specific characteristics of the datasets used.
- Future Work: The authors suggest that their findings encourage the development of more affordable FIR cameras and their integration into ADAS. They also note that if other functionalities like traffic sign recognition are needed, combining visible and FIR systems might be more beneficial. Further research could explore more sophisticated fusion techniques beyond simple concatenation.
Summary updated