Real-Time Driving Scene Semantic Segmentation

Wenfu Wang, Yongjian Fu, Zhijie Pan, Xi Li, Yueting ZhuangIEEE Access 2020 · DOI

Introduction

In the rapidly evolving field of autonomous driving, the ability of a vehicle to perceive and understand its environment at a pixel level—known as semantic segmentation—is paramount. However, deploying these complex computer vision models on embedded hardware within a vehicle presents a significant challenge: balancing the high computational demand for accuracy with the necessity for real-time processing speeds. Researchers from Zhejiang University have introduced DSNet (Driving Segmentation Network), a lightweight architecture designed to bridge this gap, achieving over 100 frames per second (FPS) while maintaining high precision, particularly for small and critical objects.

The Challenge: Speed vs. Precision

Current semantic segmentation models typically fall into two categories. Heavyweight models, such as PSPNet, offer exceptional accuracy but are computationally expensive, making them unsuitable for the split-second decision-making required in high-speed driving. Conversely, lightweight models like ENet prioritize speed but often suffer from a significant drop in accuracy.

A primary issue in urban driving scenes is class imbalance. Large, frequent categories like "road" or "sky" dominate the training data, often drowning out the learning signal for smaller, "hard" objects such as traffic signs, poles, or distant pedestrians. In a real-world driving scenario, failing to detect a small cyclist is far more dangerous than misidentifying a patch of sky.

The DSNet Architecture

To address these challenges, the authors developed DSNet, inspired by the efficiency of ShuffleNet V2. The architecture utilizes an asymmetric encoder-decoder structure:

  1. Encoder: A deep encoder extracts complex features using "Basic Units" that employ channel splitting and shuffling. This technique allows for high feature richness while significantly reducing computational costs.
  2. Dilated Convolutions: To expand the receptive field—allowing the model to see more of the scene without increasing the number of parameters—DSNet incorporates dilated convolutions. To prevent the "gridding effect" (checkerboard artifacts) often seen in standard dilated layers, the researchers implemented a Hybrid Dilated Convolution (HDC) scheme.
  3. Decoder: A lightweight decoder is used to restore the image resolution, ensuring the final output is sharp and spatially accurate.

Object Weighted Focal Loss (OWFL)

The core innovation of DSNet lies in its training strategy, specifically the Object Weighted Focal Loss (OWFL). Recognizing that architecture alone cannot solve the class imbalance problem, the researchers designed OWFL to force the network to "pay more attention" to rare and small objects.

OWFL combines two distinct weighting mechanisms:

  • Frequency Weight: Adjusts the loss based on the statistical frequency of an object's appearance in the dataset.
  • Order-of-Magnitude Weight: Specifically boosts the importance of rare objects, ensuring that even if an object occupies only a few pixels, its correct classification contributes significantly to the model's learning.

Additionally, a Semantic Encoding Loss (SEL) term was introduced to provide global context, helping the model understand the overall scene composition and reducing "impossible" misclassifications.

Experimental Results and Performance

The model was rigorously evaluated on the Cityscapes dataset, a standard benchmark for urban driving scenes. DSNet achieved a mean Intersection-over-Union (mIoU) of 69.3% on the test set.

Key performance metrics include:

  • Real-Time Speed: The model processes images at 100+ FPS on an NVIDIA 1080Ti at a resolution of 640 × 360.
  • Efficiency: With only 0.9 million parameters, DSNet is approximately 180 times lighter than heavyweight models like PSPNet.
  • Small Object Detection: DSNet outperformed ERFNet, a leading real-time model, by 2.7 percentage points on minor and difficult object classes, demonstrating the effectiveness of the OWFL strategy.

Conclusion

DSNet demonstrates that high-speed performance does not have to come at the cost of safety-critical accuracy. By combining an efficient architecture with a specialized loss function that prioritizes difficult classes, the researchers have created a robust solution for real-time driving scene understanding. Future work will focus on domain adaptation to ensure the model remains reliable across diverse weather conditions and geographic locations.

Summary updated

BibTeX
@article{wang2020real,
  author = {Wenfu Wang and Yongjian Fu and Zhijie Pan and Xi Li and Yueting Zhuang},
  title = {{Real-Time Driving Scene Semantic Segmentation}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.2975640},
}
Figure