Image Segmentation in Video Sequences: A Probabilistic Approach

Nir Friedman, Stuart RussellUAI 1997 · OpenReview

TL;DR

This paper introduces a novel probabilistic approach to segmenting moving objects in video sequences by modeling each pixel's appearance as a mixture of Gaussian distributions. This method effectively handles challenges like slow-moving objects and shadows, which plague traditional background subtraction techniques, by learning per-pixel models and using an efficient, incremental Expectation-Maximization (EM) algorithm for real-time performance.

What Problem Does This Paper Address?

The paper tackles the long-standing challenge of accurately identifying and tracking moving objects in video sequences, particularly in traffic surveillance. Traditional "background subtraction" methods, which rely on comparing current frames to a static or time-averaged background, struggle with several real-world issues:

  • Shadows: Moving objects cast shadows that are often misclassified as part of the object itself, leading to inaccurate segmentation and counting.
  • Slow-Moving or Stationary Objects: When objects move slowly or stop, they gradually corrupt the background model, making them indistinguishable from the background.
  • Lighting Changes: Variations in lighting over time can also lead to false positives or negatives.

These limitations hinder the reliability of systems that depend on accurate motion detection.

Methods

The core innovation of this paper is a probabilistic per-pixel classification approach. Instead of a simple binary "background" or "foreground" classification, the authors propose modeling each pixel's appearance over time as a mixture of Gaussian distributions, representing different states:

  • Road/Background: The typical appearance of the pixel when it's part of the static background.
  • Shadow: The appearance of the pixel when it's in the shadow of a moving object.
  • Vehicle/Moving Object: The appearance of the pixel when it's part of a moving object.

Key aspects of their method include:

  • Mixture-of-Gaussians Models: Each pixel's intensity or RGB values are modeled as a weighted sum of Gaussian distributions, capturing the variability within each class.
  • Unsupervised Learning: The models are learned automatically from the video data without requiring pre-labeled ground truth for each pixel.
  • Expectation-Maximization (EM) Algorithm: The standard EM algorithm is adapted to learn the parameters of these mixture models.
  • Incremental EM: To achieve real-time performance, a crucial modification is made to the EM algorithm. Instead of reprocessing the entire dataset for each update, the incremental version updates the model parameters as each new frame arrives, making it suitable for streaming video. This is achieved by continuously accumulating sufficient statistics.
  • Heuristic Component Labeling: Since the EM algorithm doesn't inherently label the mixture components (e.g., which Gaussian is "road" vs. "shadow"), a heuristic approach is used to assign these labels based on expected properties (e.g., shadows are typically darker than the road).

Datasets

The paper primarily uses video sequences from the Roadwatch traffic surveillance project at Berkeley. Specific details about the dataset's size or composition (e.g., number of frames, resolution) are not explicitly detailed beyond its application context. The authors also refer to general image data for illustrating pixel distributions.

Key Insights and Ideas

  • Per-Pixel Probabilistic Models are Powerful: The fundamental insight is that modeling each pixel's behavior probabilistically, rather than treating all pixels uniformly or relying on a global background, is key to overcoming the limitations of traditional methods.
  • Shadows and Slow-Moving Objects are Solvable with Better Models: The paper demonstrates that by explicitly modeling the appearance of pixels in shadow and allowing models to adapt to slow-moving objects, these persistent problems can be effectively mitigated.
  • Incremental EM Enables Real-Time Application: The adaptation of the EM algorithm to an incremental, online version is critical for practical, real-time video processing, a significant engineering achievement.
  • Heuristics for Component Labeling are Necessary but Sufficient: While the EM algorithm learns the statistical components, assigning semantic meaning (road, shadow, vehicle) requires a heuristic step, which the authors show can be effective.

Key Results

The paper presents qualitative results demonstrating the effectiveness of their approach:

  • Improved Shadow Handling: Figures 4(b) and 4(c) show that shadows are correctly identified and can be replaced with the corresponding road value, leading to cleaner segmentation.
  • Better Handling of Slow-Moving Traffic: Figure 2 illustrates how traditional background subtraction fails with slow-moving traffic (corrupted background), while the proposed method is expected to perform better.
  • Cleaner Vehicle Masks: Figure 4(d) shows a "mask showing pixels classified as vehicles" that is described as "much cleaner than that obtained by background subtraction."
  • Real-Time Potential: The incremental EM algorithm is stated to be capable of running in real time on a suitable hardware platform (e.g., multiple DSP chips).

Quantitative results (e.g., specific accuracy improvements or speed metrics) are not detailed in the provided text, but the authors state they are beginning experiments to quantify these improvements.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Initialization and Labeling: The heuristic labeling process might not work well in extreme lighting conditions. They suggest exploring MAP or penalty-based EM for more robust initialization and labeling.
  • Ignoring Spatial and Temporal Contiguity: The current pixel-based approach doesn't explicitly enforce that neighboring pixels or consecutive frames should have consistent classifications. Future work could incorporate Markov models for temporal contiguity and Markov networks for spatial contiguity.
  • Higher-Level Models: The system could be integrated with higher-level dynamic belief network models for predicting vehicle behavior, detecting events like stalled vehicles, or handling moving cameras.
  • RGB vs. Intensity: While they mention experiments with RGB models to improve detection of darker vehicles, the full impact of this is still under investigation.
  • Exponential Forgetting for Incremental EM: They are experimenting with incorporating exponential forgetting into the incremental EM process to prevent the accumulation of outdated statistics and improve long-term performance.

Summary updated

Abstract

"Background subtraction" is an old technique for finding moving objects in a video sequence for example, cars driving on a freeway. The idea is that subtracting the current image from a timeaveraged background image will leave only nonstationary objects. It is, however, a crude approximation to the task of classifying each pixel of the current image; it fails with slow-moving objects and does not distinguish shadows from moving objects. The basic idea of this paper is that we can classify each pixel using a model of how that pixel looks when it is part of different classes. We learn a mixture-of-Gaussians classification model for each pixel using an unsupervised technique- an efficient, incremental version of EM. Unlike the standard image-averaging approach, this automatically updates the mixture component for each class according to likelihood of membership; hence slow-moving objects are handled perfectly. Our approach also identifies and eliminates shadows much more effectively than other techniques such as thresholding. Application of this method as part of the Roadwatch traffic surveillance project is expected to result in significant improvements in vehicle identification and tracking.
BibTeX
@inproceedings{friedman1997image,
  author = {Nir Friedman and Stuart Russell},
  title = {{Image Segmentation in Video Sequences: A Probabilistic Approach}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {1997},
  url = {https://openreview.net/forum?id=GWHaai5QaD},
}
Figure