Towards unsupervised semantic segmentation of street scenes from motion cues

Hajar Sadeghi Sokeh, Stephen GouldIVCNZ 2012 · DOI

TL;DR

This paper introduces a novel unsupervised method for semantic segmentation of street scenes using only motion cues from video. By analyzing pixel-level motion and applying a self-calibration technique to account for perspective distortion, the approach effectively identifies meaningful regions like roads, sidewalks, and parking spots without requiring manual annotations.

What Problem Does This Paper Address?

Semantic scene segmentation, the task of labeling different regions in an image or video with their semantic meaning (e.g., road, car, building), is crucial for many AI applications. While supervised methods achieve high accuracy, they demand large amounts of manually annotated data, which is time-consuming and expensive to create. This paper tackles the challenge of performing this segmentation in an unsupervised manner, specifically for dynamic street scenes captured by a fixed camera. The core problem is to extract meaningful semantic information from video sequences without prior knowledge of object classes or their locations, relying solely on the inherent motion patterns within the scene. Existing methods often struggle with issues like perspective distortion, shadows, and the need for complex object tracking, which this paper aims to overcome.

Methods

The proposed method is built upon four key modules:

  1. Motion Detection: This module uses optical flow (specifically, the method by Sun et al. [16]) to estimate the magnitude and direction of movement for each pixel. To obtain cleaner object boundaries, it combines optical flow with frame differencing.
  2. Shadow Removal: Dynamic shadows can interfere with motion detection. The paper employs a simplified version of the "Ratio Edge" method [19] to identify and remove shadows, ensuring that only actual object motion is considered.
  3. Feature Extraction: The core features extracted are:
  4. The size of objects moving over each pixel.
  5. The magnitude of optical flows on each pixel.
  6. The direction of optical flows on each pixel. Additionally, color information (RGB and Lab spaces) is used as a complementary feature.

  7. Perspective Distortion Removal: A significant challenge in analyzing motion from video is perspective distortion, where the perceived size and speed of an object change with its distance. The authors introduce a simple self-calibration method that uses the perceived size change of a single object (assumed to have constant height) across two frames to normalize features, effectively removing perspective effects without requiring explicit camera calibration parameters.

  8. Segmentation (Clustering): The extracted features (motion and color) are quantized into discrete "visual words." Latent Dirichlet Allocation (LDA) [2] is then applied to these visual words, treating each pixel as a "document" and the entire video as a "corpus." LDA models the underlying "topics" (semantic regions) within the scene, assigning each pixel to a topic based on its feature distribution.

Datasets

The paper evaluates its method on the VIRAT video dataset [1]. Specifically, they use two videos from this dataset, described as "crowded streets and parking lots," each containing approximately 1300 frames. They sample every 10th frame for efficiency. The authors also mention creating a ground truth annotation for all pixels in the videos for quantitative evaluation.

Key Insights and Ideas

  • Motion as a Primary Cue: The paper strongly advocates for motion as a powerful, often underutilized, cue for unsupervised semantic segmentation, especially in dynamic scenes where color and texture can be ambiguous (e.g., roads and sidewalks looking similar).
  • Pixel-Level Independence: By treating each pixel independently and analyzing its motion characteristics, the method avoids the computational complexity and potential errors associated with tracking entire moving blobs across frames.
  • Self-Calibration for Perspective: The proposed method for removing perspective distortion using object size changes across frames is a clever way to achieve camera-independent features without needing explicit camera calibration parameters, which are often unavailable or difficult to obtain.
  • LDA for Semantic Topics: Applying LDA to pixel-level features is an innovative way to discover latent semantic "topics" or regions within the scene, effectively clustering pixels into meaningful categories based on their motion and appearance patterns.

Key Results

  • The method successfully segments street scenes into categories like roads, sidewalks, cross-roads, and parking spots.
  • Quantitative evaluation using F1-measure shows that their LDA-based approach generally outperforms Gaussian Mixture Models (GMM) as a standard clustering method, especially when using motion and Lab color features.
  • The segmentation results using motion and Lab features are noted to agree "fairly well" with the ground truth annotations.
  • Qualitative results (Figure 8) demonstrate the ability to distinguish between different road segments, sidewalks, and parking areas, though some challenges remain for less frequent events like parking.

Limitations and Future Work

  • Parking Spot Detection: The paper acknowledges that parking spots are difficult to detect until a vehicle is actually parked there. This suggests that static features or longer-term observations might be needed for robust parking spot identification.
  • Noisy Parking Lots: In the second video, the presence of vehicles and people moving across vacant parking spaces made those areas "much noisier," impacting segmentation.
  • Stop Place Ambiguity: Stop places on roads were sometimes clustered into two different regions due to the sequential stopping behavior of multiple vehicles.
  • Future Work: The authors express a desire to build on this work to gain a better understanding of activities by linking motion behavior with location within a scene. This implies a move towards activity recognition or more complex scene understanding tasks.

Summary updated

BibTeX
@inproceedings{sokeh2012towards,
  author = {Hajar Sadeghi Sokeh and Stephen Gould},
  title = {{Towards unsupervised semantic segmentation of street scenes from motion cues}},
  booktitle = {IVCNZ},
  year = {2012},
  doi = {10.1145/2425836.2425884},
}
Figure