Object Recognition from very few Training Examples for Enhancing Bicycle Maps

Christoph Reinders, Hanno Ackermann, Michael Ying Yang, Bodo Rosenhahn

TL;DR

This paper introduces a clever pipeline for detecting cyclist-specific traffic signs using an average of just 15 training examples per class. By combining the feature-learning power of CNNs with the small-data robustness of Random Forests—and then mapping that forest back into a Fully Convolutional Network—the authors achieve high-speed object detection and map localization without the need for massive labeled datasets.

What Problem Does This Paper Address?

While autonomous vehicle research has produced massive labeled datasets (like KITTI or Cityscapes), the same cannot be said for cycling infrastructure. Cyclists face unique challenges: different viewpoints, specific traffic signs (e.g., "bicycle boulevards"), and routes prohibited to cars.

Collecting raw image data via crowdsourcing (smartphones on handlebars) is easy, but labeling thousands of images is prohibitively expensive. Standard deep learning models like Faster R-CNN typically require thousands of examples to generalize well. The authors set out to build a system that can recognize these niche signs using "extremely little data" to help enhance digital bicycle maps.

Methods

The authors propose a multi-stage hybrid approach that bridges the gap between classical machine learning and deep learning:

  1. Feature Learning: They first train a Convolutional Neural Network (CNN) on a large, publicly available dataset (GTSRB) to learn general representations of traffic signs.
  2. Random Forest (RF) Classification: They use the pre-trained CNN as a feature extractor. These features are fed into a Random Forest. RFs are chosen because they are significantly more robust than deep neural networks when training data is scarce.
  3. RF-to-NN Mapping: To make the system efficient for full-image inference, they map the trained Random Forest into a two-hidden-layer neural network.
  4. The first layer evaluates "split rules" (decisions).
  5. The second layer determines "leaf membership."
  6. Fully Convolutional Transformation: The entire pipeline (CNN features + Mapped RF) is converted into a Fully Convolutional Network (FCN). This allows the model to process images of any size and produce a probability map in a single forward pass, rather than sliding a window pixel-by-pixel, which is computationally expensive.
  7. Localization: Finally, they integrate GPS and heading data from the smartphone sensors to project detected signs onto a global map using the haversine formula and pinhole camera geometry.

Datasets

  • GTSRB (German Traffic Sign Recognition Benchmark): Used for initial feature learning (39,209 training examples).
  • Crowdsourced Cyclist Dataset: A new dataset collected via a custom smartphone app. It contains over 40,000 images, but only 297 bounding boxes were manually annotated across 10 classes (averaging ~15-30 samples per class, with one class having as few as 4).

Key Insights and Ideas

  • The "Best of Both Worlds" Strategy: Deep learning is great for features, but bad at small-data classification. Random Forests are great at small-data classification but hard to deploy as efficient "sliding window" detectors. Mapping the RF to an FCN solves both problems.
  • Part-Based Boosting: The authors noticed that with very few examples, the network often learns to recognize parts of signs (like the bicycle symbol) rather than the whole sign. They utilized this by defining a "list of parts" for each class; if a part is detected within a larger bounding box, they boost the detection score.
  • Adaptive Filtering: To handle the massive influx of crowdsourced data, they implemented a filter that only keeps images where the cyclist is slowing down or accelerating (indicating potential intersections or obstacles), reducing the data volume by a factor of 5 before processing.

Key Results

  • Classification Accuracy: On image patches, the RF classifier achieved 99.3% accuracy.
  • Detection Performance: The system achieved a mean Average Precision (mAP) of 0.686. For some classes, like "Bicycle Road," the AP was as high as 0.994.
  • Inference Speed: Converting the RF to an FCN reduced processing time from over 10 hours per image (using a standard patch-based RF) to just 6.08 seconds for 8 different scales.
  • Localization: The median localization error on the map was 6.76 meters, which is sufficient for enhancing map metadata like OpenStreetMap.

Limitations and Future Work

  • Extreme Few-Shot Failure: Performance dropped significantly on classes with fewer than 10 examples (e.g., class 1000-32 with only 4 samples had an AP of 0.130).
  • GPS Dependency: The localization accuracy is heavily tied to the smartphone's GPS precision. In "urban canyons" with tall buildings, the error increases.
  • Future Work: The authors suggest using "multiple observations" (detecting the same sign from different frames/angles) to triangulate a more precise location and filter out false positives.

Summary updated

Abstract

In recent years, data-driven methods have shown great success for extracting information about the infrastructure in urban areas. These algorithms are usually trained on large datasets consisting of thousands or millions of labeled training examples. While large datasets have been published regarding cars, for cyclists very few labeled data is available although appearance, point of view, and positioning of even relevant objects differ. Unfortunately, labeling data is costly and requires a huge amount of work. In this paper, we thus address the problem of learning with very few labels. The aim is to recognize particular traffic signs in crowdsourced data to collect information which is of interest to cyclists. We propose a system for object recognition that is trained with only 15 examples per class on average. To achieve this, we combine the advantages of convolutional neural networks and random forests to learn a patch-wise classifier. In the next step, we map the random forest to a neural network and transform the classifier to a fully convolutional network. Thereby, the processing of full images is significantly accelerated and bounding boxes can be predicted. Finally, we integrate data of the Global Positioning System (GPS) to localize the predictions on the map. In comparison to Faster R-CNN and other networks for object recognition or algorithms for transfer learning, we considerably reduce the required amount of labeled data. We demonstrate good performance on the recognition of traffic signs for cyclists as well as their localization in maps.
BibTeX
@inproceedings{reinders2018object,
  author = {Christoph Reinders and Hanno Ackermann and Michael Ying Yang and Bodo Rosenhahn},
  title = {{Object Recognition from very few Training Examples for Enhancing Bicycle Maps}},
  booktitle = {Intelligent Vehicles Symposium},
  year = {2018},
  doi = {10.1109/ivs.2018.8500469},
}
Figure