Zero-Shot Traffic Sign Recognition Based on Midlevel Feature Matching

Yaozong Gan, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper introduces a novel "zero-shot" traffic sign recognition method that bypasses the need for extensive training data by matching mid-level features extracted from CNNs. This approach allows for effective recognition of traffic signs from different countries without retraining, making it more adaptable and efficient for real-world applications.

What Problem Does This Paper Address?

Traffic sign recognition is crucial for advanced driver assistance systems (ADAS) and road safety. However, existing methods, primarily based on Convolutional Neural Networks (CNNs), require large, meticulously curated datasets for training. This is problematic because:

  1. Data Acquisition is Costly: Collecting and annotating vast numbers of traffic sign images is time-consuming and expensive.
  2. Global Variability: Traffic signs differ significantly across countries, necessitating retraining or fine-tuning for each new region.
  3. New Sign Introduction: When new traffic sign categories emerge, existing models need to be updated.

The paper aims to address these limitations by developing a method that can recognize traffic signs without prior training on specific categories, a concept known as "zero-shot recognition."

Methods

The core of the proposed method is Traffic Sign Matching (TSM) using mid-level features extracted from pre-trained CNNs. Here's a breakdown:

  1. Traffic Sign Detection (TSD): First, the system detects and localizes traffic signs within original road images. This is achieved using a Vision Transformer Adapter (ViT-Adapter), an innovative approach for object identification, followed by contour detection to precisely outline the sign boundaries.
  2. Mid-level Feature Extraction: Instead of using the final output of a CNN (which might be too abstract) or the initial layers (which might be too generic), the method focuses on mid-level layers of pre-trained CNNs (like ResNet-50, DenseNet-121, and EfficientNet-B0). These layers are found to capture a rich blend of semantic information (what the sign represents) and shape/color details, offering a robust representation. The CNNs are pre-trained on ImageNet, but no further fine-tuning is performed for the traffic sign recognition task itself.
  3. Zero-Shot Matching: The extracted mid-level features from a target traffic sign image are then compared against the mid-level features of template traffic signs from a standardized database. The dissimilarity (calculated using Euclidean distance) between these feature vectors determines the match. The system identifies the template sign with the lowest dissimilarity as the recognized sign. This process bypasses the need for any training data specific to the target traffic sign categories.

Datasets

The proposed method was evaluated on two distinct datasets:

  1. German Traffic Sign Recognition Benchmark (GTSRB): A widely used dataset comprising 1,213 traffic sign images across 43 classes. The authors used 43 distinct classes as template signs and the remaining images as target signs.
  2. Sapporo Urban Road Dataset: A real-world dataset collected from urban road images in Sapporo City, Japan. This dataset contained 71 images representing 18 distinct traffic sign types used as target signs, with 111 categories of Japanese traffic sign templates.

Key Insights and Ideas

  • Mid-level Features are Key: The paper's central hypothesis, and a key insight, is that mid-level features from CNNs strike an optimal balance. They retain crucial visual details (shape, color) while also capturing semantic meaning, making them highly effective for matching without explicit training.
  • Zero-Shot is Achievable: The research demonstrates that robust traffic sign recognition can be achieved without the traditional requirement of large, labeled training datasets for each sign category.
  • Adaptability Across Borders: By relying on feature matching rather than category-specific training, the method inherently supports recognition of traffic signs from different countries, provided a template database exists.
  • ViT-Adapter for Detection: The use of ViT-Adapter for the initial traffic sign detection step shows promise in accurately localizing signs, even in complex road scenes.

Key Results

The proposed mid-level feature matching TSM method demonstrated strong performance on both datasets:

  • GTSRB Dataset: The method achieved significantly higher Top-k accuracy compared to traditional methods like HOG and SIFT. For instance, using ResNet-50, the mid-level features achieved a Top-1 accuracy of 0.521, Top-5 of 0.781, and Top-10 of 0.930, outperforming other layers and traditional methods.
  • Sapporo Urban Road Dataset: Similar improvements were observed. With DenseNet-121, the mid-level features yielded a Top-1 accuracy of 0.817, Top-5 of 0.873, and Top-10 of 0.915, again surpassing other layers and baseline methods.
  • Robustness: The method showed robustness in handling blurred or ambiguous traffic signs, as demonstrated in Figure 4.
  • Efficiency: While not the primary focus, the computation time per target image was found to be reasonably efficient, ranging from ~1.15s to ~5.35s depending on the CNN architecture and dataset complexity.

Limitations and Future Work

  • Template Database Dependency: The performance is inherently tied to the quality and comprehensiveness of the template traffic sign database.
  • Computational Cost: While efficient, matching against a very large number of templates could increase computation time.
  • Future Directions:
    • Real-time Video Processing: Extending the method for dynamic traffic sign recognition in video streams.
  • Model Architecture Refinement: Further optimizing CNN architectures for efficiency and accuracy.
  • Template Database Enhancement: Incorporating more diverse traffic sign variations and simulating challenging conditions (rotation, blur) in templates.
  • Handling Ambiguity: Developing strategies to set dissimilarity thresholds for more robust final sign determination, especially in adverse conditions.
  • Cross-Dataset Generalization: Further validation on datasets from more diverse regions and countries.

Summary updated

Abstract

Traffic sign recognition is a complex and challenging yet popular problem that can assist drivers on the road and reduce traffic accidents. Most existing methods for traffic sign recognition use convolutional neural networks (CNNs) and can achieve high recognition accuracy. However, these methods first require a large number of carefully crafted traffic sign datasets for the training process. Moreover, since traffic signs differ in each country and there is a variety of traffic signs, these methods need to be fine-tuned when recognizing new traffic sign categories. To address these issues, we propose a traffic sign matching method for zero-shot recognition. Our proposed method can perform traffic sign recognition without training data by directly matching the similarity of target and template traffic sign images. Our method uses the midlevel features of CNNs to obtain robust feature representations of traffic signs without additional training or fine-tuning. We discovered that midlevel features improve the accuracy of zero-shot traffic sign recognition. The proposed method achieves promising recognition results on the German Traffic Sign Recognition Benchmark open dataset and a real-world dataset taken from Sapporo City, Japan.
BibTeX
@inproceedings{gan2023zero,
  author = {Yaozong Gan and Guang Li and Ren Togo and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Zero-Shot Traffic Sign Recognition Based on Midlevel Feature Matching}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=tsPbNo59oSs},
}
Figure