Introduction
Reliable semantic segmentation is a cornerstone of autonomous vehicle navigation, requiring the precise categorization of every pixel in a scene. While standard RGB cameras provide high-resolution texture and color information, their performance degrades significantly in adverse conditions such as low light, heavy fog, or intense glare. Thermal cameras, specifically Long-Wave Infrared (LWIR), offer a complementary solution by detecting heat signatures, which remain visible regardless of lighting conditions. However, thermal images often lack the fine-grained detail necessary for comprehensive scene understanding.
The fusion of RGB and thermal data (RGBT) presents a promising path forward, yet it faces two critical challenges: spatial misalignment due to sensor parallax and varying reliability between modalities. DooDLeNet addresses these issues through a specialized dual-stream architecture designed to dynamically weight and align features from both sensors.
Architecture and Methodology
DooDLeNet utilizes a "Double DeepLab" framework, extending the popular DeepLabV3+ architecture into a multi-modal system. Unlike early-fusion approaches that merge raw data, DooDLeNet maintains separate encoder-decoder paths for the color and thermal streams. This modality specialization ensures that the network learns distinct features—such as texture from RGB and heat from LWIR—without premature interference.
The core innovation lies in the fusion stage, which employs two novel weighting mechanisms:
- Confidence Weighting (C): The model generates preliminary segmentation maps for each stream and calculates a confidence score. This allows the system to down-weight unreliable features—for instance, suppressing the RGB stream in nighttime scenarios—while prioritizing the more certain modality.
- Matching Correlation Weighting (M): To mitigate the effects of sensor misalignment, a correlation layer identifies spatial correspondences between the RGB and thermal feature maps. This internal "soft-alignment" ensures that features are correctly associated before final processing.
Finally, these weighted features are concatenated and passed to a shared decoder. This decoder utilizes Atrous Spatial Pyramid Pooling (ASPP) to capture multi-scale contextual information, producing the final high-resolution segmentation mask.
Experimental Setup and Datasets
The performance of DooDLeNet was rigorously evaluated using the Multi-spectral Fusion (MF) dataset. This benchmark consists of 1,569 synchronized RGB-thermal image pairs captured in urban environments, split between daytime (820 pairs) and nighttime (749 pairs) conditions. The dataset includes eight object classes critical for urban navigation, such as pedestrians, vehicles, and road obstacles.
Results and Discussion
DooDLeNet achieved state-of-the-art results on the MF dataset, reaching a mean Intersection over Union (mIoU) of 57.3%. This performance surpasses previous benchmarks, including FEANet (55.3%) and RTFNet-152 (53.2%).
A key highlight of the model is its efficacy in pedestrian detection, achieving a 72.2% IoU for the "Person" category. Qualitative analysis confirms that DooDLeNet maintains robust performance in scenarios where pedestrians are virtually invisible to RGB sensors, leveraging the thermal stream to maintain safety-critical detections while preserving the sharp object boundaries typically provided by color cameras.
Conclusion and Future Work
DooDLeNet demonstrates that dynamic, modality-specific weighting is essential for robust RGBT fusion. While the current implementation using a ResNet-101 backbone sets a new performance standard, future research will explore more efficient backbones to optimize real-time performance. Additionally, the authors aim to test the architecture on larger, more diverse datasets to ensure its generalizability across varying geographic and climatic conditions.
Summary updated