Abstract
Semantic segmentation has made striking progress due to the success of deep convolutional neural networks. Considering the demands of autonomous driving, real-time semantic segmentation has become a research hotspot. However, few real-time RGB-D fusion semantic segmentation studies have been carried out despite the accessibility of depth information. In this paper, we propose a real-time fusion semantic segmentation network termed RFNet that effectively exploits complementary cross-modal information. Building on an efficient network architecture, RFNet is capable of running at 22Hz at full 2048x1024 resolution. Multi-dataset training is leveraged to incorporate unexpected small obstacle detection, enriching the recognizable classes required to face unforeseen hazards in the real world. On Cityscapes, our method outperforms previous state-of-the-art semantic segmenters with excellent accuracy and speed.
Introduction
Autonomous driving systems rely heavily on environment perception. While standard semantic segmentation models focus on common categories like cars and pedestrians, they often struggle with "unexpected" obstacles—debris, stones, or fallen cargo—that do not fit into standard classes. These small objects are responsible for significant safety risks. RFNet addresses this by intelligently fusing RGB images with depth data to identify irregular road hazards while maintaining the real-time performance necessary for vehicle deployment.
The Challenge of Unexpected Obstacles
Detecting small road hazards presents two primary challenges:
- Sensor Limitations: RGB cameras struggle with small objects that blend into road textures (e.g., a gray stone on gray asphalt). Depth sensors provide geometric cues, but fusing this data without sacrificing speed is difficult.
- Dataset Gaps: Popular datasets like Cityscapes lack an "unexpected obstacle" class. Conversely, specialized datasets like Lost and Found have labeling schemes that often conflict with standard datasets, making joint training complex.
RFNet Architecture and Fusion
RFNet utilizes a dual-branch encoder-decoder architecture. Two parallel ResNet-18 branches extract features from RGB and depth images independently. The core innovation is the Attention Feature Complementary (AFC) Module. Instead of simple addition, the AFC uses Squeeze-and-Excitation (SE) blocks to identify and weight the most informative channels. This allows the network to prioritize depth information when the RGB signal is ambiguous, such as when an object's color matches the road.
Following the encoder, an efficient decoder inspired by SwiftNet uses lightweight upsampling and skip connections to restore resolution quickly. This design ensures the computational overhead remains low enough for real-time applications.
Multi-Dataset Training Strategy
To handle conflicting labels between Cityscapes (19 classes) and Lost and Found (small obstacles), the authors designed a custom loss function. It uses a hyperparameter to ignore "background" or "free-space" labels from one dataset when they would contradict the specific object labels of the other. This allowed the model to learn 20 distinct classes (19 standard + 1 obstacle) without requiring manual relabeling of thousands of images.
Experimental Results
RFNet achieves a balance of speed and accuracy that is highly competitive for autonomous driving:
- Speed: The network runs at 22Hz at 2048×1024 resolution on a single RTX 2080Ti. At half resolution, it reaches 41.6Hz.
- Accuracy: On the Cityscapes validation set, it achieved 72.5% mIoU, outperforming RGB-only models and other multimodal competitors.
- Obstacle Detection: RFNet significantly improves the detection of small obstacles in the critical 0–40 meter range.
Conclusion
RFNet demonstrates that depth information is a critical indicator for road obstacles, providing geometric cues that RGB alone lacks. By combining an efficient fusion module with a clever multi-dataset training strategy, the authors have created a robust system capable of identifying unforeseen hazards in real-time. Future work will focus on improving long-range detection where depth data quality typically degrades.
Summary updated