Describing Visual Scenes using Transformed Dirichlet Processes

Erik B. Sudderth, Antonio Torralba, William T. Freeman, Alan S. WillskyNeurIPS 2005 · OpenReview

Overview

The authors introduce the Transformed Dirichlet Process (TDP), a hierarchical probabilistic model designed to discover and localize an unknown number of objects within a visual scene. By extending the Hierarchical Dirichlet Process (HDP) to include stochastic transformations, the TDP can share spatial "object templates" across multiple images while accounting for their varying positions. This approach significantly outperforms traditional "bag-of-features" models by explicitly capturing spatial structure.

The Problem: Missing Spatial Structure

In computer vision, learning to recognize and locate objects with minimal supervision remains a significant challenge. Existing methods generally fall into two categories, both with notable limitations:

  1. Discriminative models: These achieve high accuracy but require massive amounts of manually labeled data, making them expensive to train.
  2. Generative "Bag-of-Features" models (e.g., Latent Dirichlet Allocation - LDA): These can learn without supervision but completely ignore spatial structure. They treat an image like a jumbled pile of visual parts, losing the geometric relationships that define an object.

Furthermore, most models struggle to handle uncertainty in the number of object instances. They either assume a single object per image or require a fixed, pre-defined number of parts. The authors sought a way to model scenes where the number of entities (like cars, buildings, or pedestrians) is unknown a priori, and where the spatial arrangement of features is crucial for recognition.

Comparison of clustering methods showing Constellation, LDA, and Transformed DP.

Methods: The Transformed Dirichlet Process

The core innovation of this paper is the Transformed Dirichlet Process. To understand its mechanics, it helps to trace its lineage:

  • Dirichlet Process (DP): A Bayesian non-parametric tool for clustering where the number of clusters is not fixed but grows organically with the data.
  • Hierarchical DP (HDP): An extension that allows multiple groups (e.g., different images) to share the exact same set of clusters. However, HDP assumes these clusters are identical across all groups.
  • The TDP Twist: In visual scenes, an object (a cluster of features) rarely appears in the exact same location across different images. The TDP addresses this by allowing shared mixture components to undergo a stochastic transformation (such as a 2D translation) before being reused in a new image.
Graphical models comparing LDA, Hierarchical DP, Transformed DP, and Visual Scene TDP.

Within this framework, mixture components describe the spatial structure of visual features in an object-centered coordinate frame, while transformations model the specific position of that object in a particular image. Inference is handled via a specialized Gibbs sampler, which iteratively assigns visual features to "tables" (representing object instances) and "dishes" (representing object categories).

Datasets and Evaluation

The model was evaluated using two primary sources:

  1. Synthetic 2D Data: A dataset of bars and blobs used to demonstrate that the TDP can successfully distinguish between global cluster structures and local translations better than a standard HDP.
  2. MIT-CSAIL Street Scenes: A real-world dataset of 325 images containing buildings, cars, and roads. The training set was semi-supervised, flexibly utilizing both fully labeled images and partially segmented ones.

Key Insights and Results

  • "Style and Content" Separation: The TDP effectively separates the "content" (the spatial layout of an object's features) from the "style" or "pose" (its location in the image).
  • Beyond the Sliding Window: Unlike traditional detectors that slide a window across every pixel, the TDP performs a "global parsing" of the scene. It estimates the location and category of objects by finding the most coherent explanation for all observed features simultaneously.
  • Automatic Instance Counting: Because the model treats each object as a transformed cluster, it naturally estimates the number of instances (e.g., identifying three distinct cars) rather than just flagging "car-like pixels."
  • Performance Boost: The TDP significantly outperformed LDA in segmenting features into correct categories. The inclusion of spatial structure provided the "missing link" that appearance-only models lacked, successfully localizing multiple instances of cars in complex street scenes.

Limitations and Future Work

While powerful, the model has limitations. The authors noted textural confusion, such as foliage being mislabeled as "road" due to shared visual textures and shadow appearances. Additionally, the current TDP is context-free; it models the internal structure of an object but doesn't explicitly encode contextual relationships (e.g., cars are usually found on roads and below buildings). Future work could incorporate richer contextual cues and more complex internal object structures to improve performance in dense, cluttered environments.

Summary updated

Abstract

Motivated by the problem of learning to detect and recognize objects with minimal supervision, we develop a hierarchical probabilistic model for the spatial structure of visual scenes. In contrast with most existing models, our approach explicitly captures uncertainty in the number of object instances depicted in a given image. Our scene model is based on the transformed Dirichlet process (TDP), a novel extension of the hierarchical DP in which a set of stochastically transformed mixture components are shared between multiple groups of data. For visual scenes, mixture components describe the spatial structure of visual features in an objectcentered coordinate frame, while transformations model the object positions in a particular image. Learning and inference in the TDP, which has many potential applications beyond computer vision, is based on an empirically effective Gibbs sampler. Applied to a dataset of partially labeled street scenes, we show that the TDP's inclusion of spatial structure improves detection performance, flexibly exploiting partially labeled training images.
BibTeX
@inproceedings{sudderth2005describing,
  author = {Erik B. Sudderth and Antonio Torralba and William T. Freeman and Alan S. Willsky},
  title = {{Describing Visual Scenes using Transformed Dirichlet Processes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2005},
  url = {https://openreview.net/forum?id=HyNCAuZObr},
}
Figure