Abstract
Standard RGB-based human detection systems frequently fail in adverse environmental conditions such as total darkness, intense glare, or smoke-filled corridors. While thermal cameras and mmWave radars offer alternative sensing modalities, they are limited by low contrast and multipath interference, respectively. This paper presents UTM, a novel multimodal fusion framework that utilizes a Bayesian Feature Extractor and an uncertainty-guided fusion strategy to robustly detect humans in visually degraded environments. By explicitly modeling the uncertainty in sensor data, UTM outperforms existing state-of-the-art methods by over 8% in detection accuracy.
Introduction
Human detection is a foundational task for autonomous systems, search-and-rescue robotics, and smart surveillance. While RGB cameras provide rich semantic information, they are essentially "fair-weather" sensors. In real-world scenarios involving airborne particles (dust, smoke) or extreme lighting variations, RGB performance degrades significantly.
To address these limitations, researchers have turned to thermal cameras, which detect infrared radiation, and mmWave radars, which are unaffected by lighting and can penetrate smoke. However, these sensors introduce new challenges. Thermal imagery often lacks distinctive features when ambient temperatures approach human body temperature, resulting in "flat" temperature fields. Conversely, mmWave radar provides sparse point clouds and is susceptible to multipath effects—where signals bounce off reflective surfaces like walls, creating "ghost" detections. The UTM framework addresses these challenges by treating sensor noise not as a hindrance, but as a signal to guide the fusion process.
The UTM Framework
The UTM (Uncertainty-guided Thermal and Mmwave) architecture consists of three primary stages designed to align, extract, and fuse disparate sensor data.
1. Data Alignment and Preprocessing
To facilitate fusion, the sparse 3D radar point clouds must be projected into a 2D representation compatible with thermal imagery. Using the thermal camera's intrinsic and extrinsic parameters, the radar data is projected onto a 2D plane to create a "radar depth image." This ensures that spatial features in both modalities are perfectly aligned before feature extraction.
2. Bayesian Feature Extractor (BFE)
Traditional feature extractors use deterministic weights, which can lead to overconfident but incorrect predictions in noisy environments. The UTM framework employs a Bayesian Feature Extractor (BFE) that treats weights as distributions. By implementing Dropout during both the training and inference phases, the model performs stochastic forward passes to estimate the variance (uncertainty) of the extracted features. This allows the system to quantify the reliability of the data coming from each sensor at any given moment.
3. Uncertainty-Guided Fusion (UGF)
The core innovation of this work is the Uncertainty-Guided Fusion (UGF) module. While standard attention mechanisms typically focus on high-confidence features, UGF utilizes the variance maps generated by the BFE to identify regions of high uncertainty. By forcing the network to pay increased attention to these "hard examples" during training, the model learns to resolve ambiguities—such as distinguishing a human signature from multipath radar noise or low-contrast thermal patches.
4. Multiscale Detection Net (MDN)
The fused feature maps are processed by a Multiscale Detection Net based on the YOLOv5s architecture. This component generates final bounding boxes and class probabilities across multiple scales, ensuring robust detection of individuals at varying distances.
Experimental Evaluation
The UTM Dataset
Due to the lack of existing datasets combining portable thermal cameras and single-chip mmWave radar, the authors curated the UTM Dataset. It comprises 24,241 frames collected across 10 diverse indoor environments, including offices, kitchens, and atriums. Ground truth labels were generated using an Intel RealSense D455 RGB camera and the Detectron2 tool, then projected into the thermal/radar coordinate space.
Performance Results
UTM demonstrates significant improvements over existing multimodal approaches:
- Accuracy: The framework achieved a mAP50:95 of 0.644, representing an 8.4% improvement over the previous state-of-the-art (VA method).
- Stability: The system showed high resilience to variations in Non-Maximum Suppression (NMS) thresholds, indicating stable and precise bounding box localization.
- Efficiency: When deployed on an NVIDIA Jetson edge device, the model achieves 5-7 FPS. While optimization is required for high-speed robotics, this frame rate is viable for many monitoring and slow-moving robotic applications.
Conclusion
The UTM framework demonstrates that uncertainty-guided fusion is a powerful strategy for handling the inherent noise of thermal and radar sensors. By shifting the focus of the network toward high-uncertainty regions, the system achieves superior robustness in visually degraded environments. Future work will focus on model compression to improve inference speeds and exploring the applicability of uncertainty-guided fusion to other sensor pairs, such as LiDAR and RGB.
Summary updated