Abstract
Semantic segmentation is a critical component of autonomous vehicle perception, yet standard models often rely exclusively on 2D RGB data. This paper explores the integration of geometric (depth) and temporal (optical flow) cues to enhance segmentation performance. By evaluating various fusion strategies on synthetic and real-world datasets, we demonstrate that mid-fusion architectures significantly improve the identification of structured and dynamic objects, even when using noisy, estimated auxiliary data.
Introduction
In the context of autonomous navigation, assigning a semantic class to every pixel is vital for safe operation. While deep learning has advanced RGB-based segmentation, these models frequently struggle with objects that blend into complex backgrounds or lack distinct color features. This research addresses these limitations by incorporating depth and motion information, which provide inherent structural and dynamic context that 2D color images lack.
Methodology
The study utilizes an FCN8s architecture with a VGG encoder as the base model. The primary focus is on the Mid-Fusion strategy, where separate encoders process RGB, depth, and optical flow modalities independently. Features are extracted and then combined at the final layer of the encoder via a summation junction. This allows the network to develop specialized feature representations for each modality before integration.
Experimental Results
The proposed methods were tested on Virtual-KITTI (providing ground truth) and Cityscapes (using estimated data). On Virtual-KITTI, the transition from unimodal RGB to an RGB+Depth (RGB+D) mid-fusion model resulted in a substantial Mean Intersection over Union (mIoU) increase from 66.47% to 72.13%. Notably, the IoU for the "Truck" class nearly doubled, rising from 33.66% to 62.84%.
On the real-world Cityscapes dataset, the use of estimated depth and flow still yielded improvements. For instance, adding Farneback optical flow improved "Motorcycle" detection by 14% compared to the RGB baseline, while estimated depth (SGM) boosted "Bus" and "Train" detection by 6.3% and 9%, respectively.
Discussion and Conclusion
The results confirm that geometric and temporal cues are highly effective for segmenting driving scenes. Mid-fusion architectures prove more robust than early-fusion by allowing the model to learn modality-specific features. Crucially, the performance gains persist even when auxiliary data is imperfect—such as that produced by CNN-based estimators like monoDepth—suggesting that deep networks can effectively filter noise from estimated maps. Future work will focus on optimizing these multi-stream architectures for real-time deployment on embedded hardware.
Summary updated