Motion and Depth Augmented Semantic Segmentation for Autonomous Navigation

Hazem Rashed, Ahmad El Sallab, Senthil Kumar Yogamani, Mohamed El-HelwWorkshops @ CVPR 2019 · DOI

Abstract

Semantic segmentation is a critical component of autonomous vehicle perception, yet standard models often rely exclusively on 2D RGB data. This paper explores the integration of geometric (depth) and temporal (optical flow) cues to enhance segmentation performance. By evaluating various fusion strategies on synthetic and real-world datasets, we demonstrate that mid-fusion architectures significantly improve the identification of structured and dynamic objects, even when using noisy, estimated auxiliary data.

Introduction

In the context of autonomous navigation, assigning a semantic class to every pixel is vital for safe operation. While deep learning has advanced RGB-based segmentation, these models frequently struggle with objects that blend into complex backgrounds or lack distinct color features. This research addresses these limitations by incorporating depth and motion information, which provide inherent structural and dynamic context that 2D color images lack.

Methodology

The study utilizes an FCN8s architecture with a VGG encoder as the base model. The primary focus is on the Mid-Fusion strategy, where separate encoders process RGB, depth, and optical flow modalities independently. Features are extracted and then combined at the final layer of the encoder via a summation junction. This allows the network to develop specialized feature representations for each modality before integration.

The Mid-Fusion architecture extracts features from multiple modalities (RGB, Depth, Flow) using separate encoders before combining them at a summation junction.

Experimental Results

The proposed methods were tested on Virtual-KITTI (providing ground truth) and Cityscapes (using estimated data). On Virtual-KITTI, the transition from unimodal RGB to an RGB+Depth (RGB+D) mid-fusion model resulted in a substantial Mean Intersection over Union (mIoU) increase from 66.47% to 72.13%. Notably, the IoU for the "Truck" class nearly doubled, rising from 33.66% to 62.84%.

Quantitative results on the Virtual-KITTI dataset demonstrate significant performance gains across various classes when augmenting RGB with depth and flow.

On the real-world Cityscapes dataset, the use of estimated depth and flow still yielded improvements. For instance, adding Farneback optical flow improved "Motorcycle" detection by 14% compared to the RGB baseline, while estimated depth (SGM) boosted "Bus" and "Train" detection by 6.3% and 9%, respectively.

Discussion and Conclusion

The results confirm that geometric and temporal cues are highly effective for segmenting driving scenes. Mid-fusion architectures prove more robust than early-fusion by allowing the model to learn modality-specific features. Crucially, the performance gains persist even when auxiliary data is imperfect—such as that produced by CNN-based estimators like monoDepth—suggesting that deep networks can effectively filter noise from estimated maps. Future work will focus on optimizing these multi-stream architectures for real-time deployment on embedded hardware.

Summary updated

BibTeX
@inproceedings{rashed2019motion,
  author = {Hazem Rashed and Ahmad El Sallab and Senthil Kumar Yogamani and Mohamed El-Helw},
  title = {{Motion and Depth Augmented Semantic Segmentation for Autonomous Navigation}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00049},
}
Figure