Learning Appearance Based Models: Mixtures of Second Moment Experts

Christoph Bregler, Jitendra MalikNeurIPS 1996

Introduction

In the mid-90s, computer vision began shifting away from complex 3D geometric models toward "appearance-based" models that learned directly from 2D images. However, dominant techniques of the era, such as Principal Component Analysis (PCA) and Eigenfaces, relied heavily on global representations of the entire image.

These global methods suffered from two major flaws:

  1. Fragility: They struggled significantly with occlusions or variations that only affected a specific part of an object.
  2. Non-linearity: Real-world objects, such as vehicles, exhibit massive "in-class" variation due to different poses and shapes. This creates complex, non-linear decision boundaries that simple linear models cannot easily navigate.

To overcome these challenges, this paper introduces a robust object recognition framework that moves away from global "eigen-image" approaches toward a "localist," part-based representation. By combining a novel texture descriptor called Generalized Second Moments (GSM) with a Hierarchical Mixture of Experts (HME) architecture, the authors achieved a state-of-the-art 6.5% error rate in vehicle classification.

Methodology

The authors propose a two-stage pipeline designed to create a representation that is local enough to be robust against occlusion, yet structured enough to capture the global "essence" of an object.

1. Generalized Second Moments (GSM)

Instead of relying on hard edges or junctions—which are often noisy and unreliable—the researchers decompose the image into a 10x10 grid of overlapping tiles.

A normalized vehicle image demonstrating the local patch-based approach used for feature extraction.

For each tile, they apply a bank of multi-scale, multi-orientation Gaussian derivative filters. They then compute a covariance matrix of these filter responses. This GSM acts as a "soft" histogram of orientations and scales, capturing the texture and structure of a local patch without making binary decisions about where an edge starts or ends. This representation is shift-invariant and can be adapted to be illumination-invariant.

2. Hierarchical Mixture of Experts (HME)

To handle the non-linear nature of vehicle shapes, the authors employ an HME architecture. This operates on a "divide and conquer" strategy: a "gating network" evaluates the input and decides which "expert" (a local linear classifier) is best suited to handle that specific type of image. For example, one expert might specialize in the visual features of vans, while another focuses on sedans. The entire system is trained using the Expectation-Maximization (EM) algorithm.

Dataset and Experimental Setup

The study utilized a real-world dataset known as the Freeway Surveillance Database, consisting of 285 images captured from a camera mounted above a freeway.

Freeway Surveillance Database showing the overhead camera perspective used to capture vehicle images.

The dataset was divided into five vehicle categories: Modern Sedan, Old Sedan, Van with back-doors, Van without back-doors, and the Volkswagen Bug. During preprocessing, images were segmented based on motion and normalized to a strict 100x100 pixel resolution. Because of this normalization, the model could not rely on bounding box size or aspect ratio to "cheat"; it was forced to learn classifications based on appearance alone.

Key Insights and Results

The GSM + HME approach demonstrated a clear leap in performance over the contemporary state-of-the-art, achieving a 6.5% misclassification rate. This significantly outperformed Nearest Neighbors (1-NN) at 15.7% and Eigen-images (PCA) at 17.4%.

Several key insights emerged from the results:

  • The "Localist" Advantage: By breaking the image into a grid, the model gained substantial robustness. If a car's taillight was obscured, the "experts" analyzing the roof or tires could still contribute to a correct classification.
  • Beyond Gradients: Standard second moment matrices typically rely on simple image gradients. The "Generalized" version uses a rich, steerable, and scalable family of filters, allowing the model to capture much more nuanced local texture information.
  • Soft Decisions Preserve Uncertainty: By using a covariance matrix of filter responses rather than thresholded edges, the representation preserves uncertainty, preventing early-stage feature extraction errors from cascading through the classification system.
  • Data Thresholds: The authors noted that while HME is powerful, it requires a "critical mass" of training data. With fewer than 120 training images, simpler methods like Nearest Neighbors actually performed better. However, HME scaled much more effectively as the dataset size increased.

Limitations and Future Work

While highly accurate, the study acknowledged a few limitations. The database was relatively small (285 images), and the authors stressed the need to test on larger, more diverse datasets to truly validate the HME architecture's scalability. Additionally, the hierarchical nature of HME and the multi-filter bank approach are computationally more intensive than simple PCA.

For future work, the authors suggested applying this part-based decomposition technique to face recognition and larger vehicle databases to see if the robust, localist approach holds up across different domains of object recognition.

Summary updated

BibTeX
@inproceedings{bregler1996learning,
  author = {Christoph Bregler and Jitendra Malik},
  title = {{Learning Appearance Based Models: Mixtures of Second Moment Experts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1996},
}
Figure