Segmentation-Based Urban Traffic Scene Understanding

Andreas Ess, Tobias Mueller, Helmut Grabner, Luc Van GoolBMVC 2009 · DOI

TL;DR

This paper introduces a two-stage system for understanding urban traffic scenes, classifying road types and detecting objects like cars and pedestrians. By first creating a "meta-representation" of the scene through patch-wise classification, their method achieves better performance on object detection compared to state-of-the-art global scene descriptors.

What Problem Does This Paper Address?

Autonomous driving and advanced driver-assistance systems (ADAS) require a robust understanding of the surrounding environment. While GPS and maps provide valuable information, they have limitations, especially in dynamic urban settings where GPS can be unreliable and maps may not reflect real-time changes like construction detours. Furthermore, maps are static and don't account for dynamic traffic participants like other vehicles, pedestrians, or cyclists. This paper aims to bridge this gap by using image-based analysis to not only identify the road layout (straight, curves, intersections) but also to detect the presence of common objects, thereby providing richer contextual information for these systems.

Methods

The proposed system, termed Segmentation-Based Urban Traffic Scene Understanding (SUTSU), employs a two-stage approach:

  1. Patch-wise Scene Classification (Meta-Representation): The first stage breaks down the input image into small patches (8x8 pixels). For each patch, a set of features is extracted, including texture information derived from the Walsh-Hadamard transform and, if available, depth information. A supervised classifier (AdaBoost with decision stumps) is trained to assign one of 13 urban texture classes (e.g., road, car, building) to each patch. This creates a "meta-representation" of the scene, which is more robust to variations in lighting and color than raw image data.
  2. Scene Classification: The second stage uses the probabilities from the patch classifier to infer higher-level scene understanding. It extracts features from the patch classification output, including:
  3. Rough Layout: Hierarchical representations (downsampled probability maps) and row/column-wise mean classifier strengths to capture overall structure.
  4. Periodic Structures: Auto-correlation analysis on subsampled probability maps to detect recurring patterns like pedestrian crossings.
  5. Orientation: Orientation histograms of road markings and curbs to determine road direction. These features are then fed into another AdaBoost classifier to distinguish between 8 road types and detect the presence of objects (cars, pedestrians, pedestrian crossings). Temporal smoothing using Hidden Markov Models (HMMs) is applied to the classifier outputs for video sequences.

Datasets

The paper uses two challenging urban datasets recorded from a moving vehicle:

  • Seq. DAY: Approximately 22,500 frames, recorded during the day. This dataset is used for training and testing the scene classifier via cross-validation.
  • Seq. DUSK: Approximately 15,000 frames, recorded later the same day under dusk conditions, characterized by different lighting and more traffic. This dataset is used solely for testing.

For training the initial patch classifier, 39 images from a separate, similar training set were manually segmented into texture classes.

Key Insights and Ideas

  • Intermediate Meta-Representation is Key: The core idea is that by first segmenting the scene into basic texture classes (the meta-representation), the system can abstract away from low-level image variations (like color aberrations) and provide a more stable input for higher-level scene understanding.
  • Patch-wise Classification for Robustness: Using patch-wise classification allows the system to handle variations in scale and perspective more effectively than purely global image features.
  • Specialized Features for Objects: The inclusion of features specifically designed to capture periodic structures (like pedestrian crossings) and orientation significantly improves the detection of objects, outperforming general scene descriptors.
  • Adaptability: The meta-representation is presented as adaptable to different image conditions (e.g., dusk, rain) without needing to retrain the high-level classifier, suggesting a modular design.

Key Results

  • Object Detection Performance: The proposed SUTSU system significantly outperforms the GIST (global scene descriptor) on object classes, demonstrating the advantage of its segmentation-based approach for this task. While GIST achieves comparable accuracy for global road types (AC: 0.57 for SUTSU vs. 0.56 for GIST), SUTSU shows better performance on object detection.
  • Road Type Classification: SUTSU can distinguish basic road types (straight, curves) well, but struggles with complex junctions due to low resolution at distance.
  • Computational Performance: The mixed C/C++ and Matlab implementation takes about 1 second for patch classification and 1-2 seconds for scene classification, with potential for parallelization for real-time applications.

Limitations and Future Work

  • Junction Complexity: The system struggles with complex junction types, especially at greater distances, due to resolution limitations.
  • Object Detection Goal: The authors emphasize that their goal is not precise object localization but rather detecting the presence of objects.
  • Future Work:
    • Improve Patch Classifier: Incorporate more features like 3D points or optic flow.
  • Enhance Scene Classifier: Integrate vehicle ego-motion and trajectory information.
  • Sensor Fusion: Combine vision data with GPS map data.
  • Expand Classes: Increase the number of recognized road types and objects.
  • Application: Investigate the use of the output for object tracking and autonomous driving.
  • Dusk Conditions: Training the patch classifier on dusk conditions is suggested to alleviate issues like headlight reflections.

Summary updated

BibTeX
@inproceedings{ess2009segmentation,
  author = {Andreas Ess and Tobias Mueller and Helmut Grabner and Luc Van Gool},
  title = {{Segmentation-Based Urban Traffic Scene Understanding}},
  booktitle = {BMVC},
  year = {2009},
  doi = {10.5244/c.23.84},
}

Loading related content...

Figure