A Generative Model for Parts-based Object Segmentation

S. M. Ali Eslami, Christopher K. I. WilliamsNeurIPS 2012 · OpenReview

Introduction

In computer vision, identifying an object's bounding box is often insufficient for high-level tasks like action recognition or detailed scene understanding. To achieve a deeper understanding, systems require parts-based segmentation, where an image is partitioned into specific semantic components (e.g., separating a person into head, torso, arms, and legs).

The primary challenge in parts-based segmentation lies in the immense variability of both shape and appearance. Traditional models often rely on local neighborhood statistics, such as Markov Random Fields (MRFs), which struggle to capture global constraints. For instance, a local model might not recognize that if a person's left leg is long, their right leg is likely long too. While the original Shape Boltzmann Machine (SBM) was a breakthrough for modeling binary foreground/background shapes, it could not distinguish between different parts of the same object.

To address this, researchers have introduced the Multinomial Shape Boltzmann Machine (MSBM), a generative model that extends the SBM to represent objects as a collection of parts. By combining this hierarchical shape prior with an appearance model, the authors create a fully generative framework capable of complex object "parsing" through probabilistic inference.

Methods

The proposed framework is a joint generative model consisting of two primary components that work together to parse an image.

Joint Generative Model Pipeline

1. The Multinomial Shape Boltzmann Machine (MSBM)

The MSBM is the core innovation of this work. It extends the standard SBM by utilizing multinomial visible units. Instead of a pixel being strictly 0 (background) or 1 (foreground), it can now take one of $L+1$ labels, representing the background plus $L$ distinct object parts.

Evolution of Shape Models

To capture both fine local part details and broad global object structure, the MSBM employs a deep architecture with two layers of latent variables ($h^1, h^2$). To keep the number of parameters manageable and computationally efficient, the model uses local receptive fields and weight sharing in the first hidden layer, effectively bridging the gap between overly simplistic local models and fully connected, parameter-heavy networks.

2. Appearance Model

To complement the shape prior, the authors model the "color" or texture of parts using a mixture of $W$ Gaussians in RGB space. The underlying assumption is that while a specific part (like a "torso") might exhibit different appearances across various images (e.g., a red shirt vs. a blue jacket), these appearances can be effectively clustered into $K$ distinct classes.

3. Joint Inference

The model combines the MSBM shape prior and the Gaussian appearance model into a single joint probability distribution. Image segmentation is then treated as a probabilistic inference problem: given an observed image, what is the most likely configuration of part labels? The authors solve this using Block-Gibbs Markov Chain Monte Carlo (MCMC) sampling. Because the latent space is high-dimensional, inference is often seeded by starting multiple MCMC chains from different initial segmentations, using a scoring function to select the most physically plausible result.

Key Insights and Results

Unlike many segmentation models that learn a direct, discriminative mapping from pixels to labels, this framework is entirely generative. It learns the underlying probability distribution of what objects and their parts look like. This allows the model to "imagine" or sample realistic new object shapes that were never seen in the training set. Furthermore, because the model possesses a strong global shape prior, it naturally handles occlusion; if part of an object is hidden, the MSBM can infer the most likely shape of the occluded parts based on the visible context.

The model was evaluated on two challenging datasets:

  • Penn-Fudan Pedestrians: Consisting of 169 images annotated with 7 parts (hair, face, upper clothes, lower clothes, shoes, legs, arms). The MSBM achieved an average accuracy of 66.6% (when combined with superpixels), comparable to the then-state-of-the-art pedestrian-specific algorithm (69.5%), despite being a general-purpose model.
  • ETHZ Cars: Consisting of 139 images annotated with 6 parts (body, wheel, window, bumper, license plate, headlight). The model achieved 86.0% accuracy, nearly matching the highly specialized Implicit Shape Model (86.8%).

Qualitatively, the model produces visually coherent segmentations even in the presence of significant pose and appearance changes, successfully parsing objects into their constituent parts while respecting anatomical or structural rules.

Limitations and Future Work

While powerful, the model has a few limitations. The current appearance model relies on relatively simple pixel-wise Gaussian Mixture Models. Incorporating richer spatial structure into the appearance model—such as textures or gradients—would likely improve performance on fine structures like headlights or faces. Additionally, MCMC sampling is computationally expensive. Future work could focus on finding more efficient inference methods or better ways to initialize the generative model using fast, discriminative bottom-up approaches. Finally, the model's peak accuracy sometimes relies on a superpixel pre-processing step to enforce local smoothness, suggesting room for refinement in the pixel-wise generative process itself.

Summary updated

Abstract

The Shape Boltzmann Machine (SBM) has recently been introduced as a state-of-the-art model of foreground/background object shape. We extend the SBM to account for the foreground object's parts. Our model, the Multinomial SBM (MSBM), can capture both local and global statistics of part shapes accurately. We combine the MSBM with an appearance model to form a fully generative model of images of objects. Parts-based image segmentations are obtained simply by performing probabilistic inference in the model. We apply the model to two challenging datasets which exhibit significant shape and appearance variability, and find that it obtains results that are comparable to the state-of-the-art.
BibTeX
@inproceedings{eslami2012generative,
  author = {S. M. Ali Eslami and Christopher K. I. Williams},
  title = {{A Generative Model for Parts-based Object Segmentation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2012},
  url = {https://openreview.net/forum?id=BkbiWtb_WB},
}
Figure