Traffic Sign Detection and Recognition for Autonomous Driving in Virtual Simulation Environment

Meixin Zhu, Jingyun Hu, Ziyuan Pu, Zhiyong Cui, Liangwu Yan, Yinhai WangarXiv 2019 · OpenReview

Abstract

This study developed a traffic sign detection and recognition algorithm based on the RetinaNet. Two main aspects were revised to improve the detection of traffic signs: image cropping to address the issue of large image and small traffic signs; and using more anchors with various scales to detect traffic signs with different sizes and shapes. The proposed algorithm was trained and tested in a series of autonomous driving front-view images in a virtual simulation environment. Results show that the algorithm performed extremely well under good illumination and weather conditions. Its drawbacks are that it sometimes failed to detect object under bad weather conditions like snow and failed to distinguish speed limits signs with different limit values.

Introduction

In the development of autonomous driving systems, traffic sign recognition is a critical safety requirement. However, detecting these signs in high-resolution imagery presents a significant "needle in a haystack" challenge. In the virtual simulation environment used for this study, images are captured at a resolution of 3200 × 1800 pixels, while traffic signs often occupy as little as 50 × 50 pixels.

Standard object detection models struggle with this scale disparity. Downsampling high-resolution images to fit GPU memory constraints often causes small traffic signs to lose essential features, becoming indistinguishable. Conversely, maintaining full resolution imposes prohibitive computational costs. Additionally, virtual environments introduce complexities such as varying weather conditions, lighting changes, and dynamic obstacles like pedestrians and non-motor vehicles.

Methods

The researchers utilized RetinaNet as the base architecture. RetinaNet is a one-stage detector designed to balance speed and accuracy by employing a "Focal Loss" function, which effectively manages the extreme imbalance between background pixels and foreground objects.

To optimize RetinaNet for high-resolution traffic sign detection, two primary modifications were implemented:

  1. Strategic Image Cropping: To process the 3200 × 1800 images without losing detail, a cropping pipeline was introduced. During training, 400 × 400 sub-images centered on traffic signs were used. For testing, random cropping was applied, and results were subsequently merged. This approach allows the model to analyze signs at their native resolution.
  2. Enhanced Anchor Scales: The standard RetinaNet anchor set was expanded to include a wider range of scales (0.1 to 1.5) and three aspect ratios (1:2, 1:1, 2:1). This ensures the model has appropriate "templates" for signs of various shapes and distances.

The system's backbone consists of a ResNet-101 integrated with a Feature Pyramid Network (FPN), facilitating multi-scale feature extraction.

The RetinaNet architecture featuring a ResNet backbone and Feature Pyramid Network (FPN) for multi-scale object detection.

Datasets and Environment

The study leveraged data from a specialized competition for Traffic Sign Recognition in Virtual Simulation Environments.

  • Training Set: Approximately 14 GB of RGB images with labels for 21 classes, including parking areas, speed limits, and roundabouts.
  • Testing Set: Approximately 16 GB of RGB images.
  • Environment: A high-fidelity virtual simulation featuring diverse weather, illumination, and interference factors.
Examples of the virtual simulation environment illustrating diverse weather and lighting conditions, such as fog, rain, and clear skies.

Results and Analysis

The adapted RetinaNet model achieved a high F1 score of 0.923 on the testing dataset. The average processing speed was approximately 0.15 seconds per image (6.6 FPS) on a dual NVIDIA Tesla P100 system. Notably, the evaluation required an Intersection over Union (IoU) greater than 0.9 for successful recognition, a stringent precision requirement that the model met under standard conditions.

Key insights from the results include:

  • Resolution over Context: For small object detection, local resolution is more critical than global context. Cropping effectively "zooms in," preserving features that would be lost in full-frame resizing.
  • Anchor Optimization: Manually expanding the anchor search space to include very small scales is vital for high-precision localization in specialized tasks.

Limitations and Future Work

Despite its strong performance, the model exhibited two primary failure modes:

  1. Extreme Weather: Detection performance dropped significantly during heavy simulated snow, where visual features are obscured by environmental noise.
  2. Fine-Grained Classification: The model occasionally struggled to distinguish between different numerical values on speed limit signs (e.g., misidentifying 30 km/h as 40 km/h).

Future research will explore anchor-free detection methods to better handle scale variance and the implementation of Mask R-CNN for segmentation-based detection. Additionally, further data augmentation techniques will be investigated to improve robustness against environmental occlusion and class imbalance.

Summary updated

Abstract

This study developed a traffic sign detection and recognition algorithm based on the RetinaNet. Two main aspects were revised to improve the detection of traffic signs: image cropping to address the issue of large image and small traffic signs; and using more anchors with various scales to detect traffic signs with different sizes and shapes. The proposed algorithm was trained and tested in a series of autonomous driving front-view images in a virtual simulation environment. Results show that the algorithm performed extremely well under good illumination and weather conditions. Its drawbacks are that it sometimes failed to detect object under bad weather conditions like snow and failed to distinguish speed limits signs with different limit values.
BibTeX
@misc{zhu2019traffic,
  author = {Meixin Zhu and Jingyun Hu and Ziyuan Pu and Zhiyong Cui and Liangwu Yan and Yinhai Wang},
  title = {{Traffic Sign Detection and Recognition for Autonomous Driving in Virtual Simulation Environment}},
  year = {2019},
  eprint = {1911.05626},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.05626},
}
Figure