Object Bank: A High-Level Image Representation for Scene Classification & Semantic Feature Sparsification

Li-Jia Li, Hao Su, Eric P. Xing, L. Fei-FeiNeurIPS 2010 · OpenReview

Traditional computer vision has long relied heavily on low-level features like GIST (gradients) or SIFT (local textures) to understand images. While effective for basic tasks, these features lack semantic meaning. For instance, a dense forest mountain and a busy city street might share very similar edge distributions and color histograms, leading to classification errors.

Humans do not perceive scenes as statistical distributions of gradients; we see them as collections of recognizable objects—trees, cars, buildings, and sky. The disconnect between raw pixel data and human-perceivable meaning is known as the "semantic gap." To bridge this gap, researchers propose a high-level representation called the Object Bank (OB), which explicitly identifies the "what" and "where" of objects within a scene to achieve state-of-the-art classification.

Comparison of low-level features (GIST, SIFT) and high-level Object Bank filters.

The Object Bank Representation

The Object Bank functions as a generalized object convolution, moving beyond pixels to treat objects as the fundamental "words" of a visual scene. The methodology breaks down into several key steps:

  1. The Bank of Detectors: The authors curated a collection of 200 pre-trained generic object detectors. These include "blobby" objects (like cars and people) identified using Latent SVMs, as well as "stuff" (like sky, water, and grass) identified using texture-based classifiers.
  2. Multi-Scale Response: Every detector is run across the input image at 12 different scales to capture objects regardless of their size or distance from the camera.
  3. Spatial Pyramid Pooling: For each object at each scale, the resulting response map is divided into a three-level spatial pyramid ($1\times1$, $2\times2$, and $4\times4$ grids). The maximum response in each grid cell is recorded, preserving rough spatial information without requiring exact localization.
  4. High-Dimensional Feature Vector: This process results in a massive feature vector calculated as: Number of Objects $\times$ Number of Scales $\times$ 21 pyramid grids. For 200 objects, this yields a highly detailed, roughly 44,604-dimensional representation of the image.

Structured Sparsity and Semantic Compression

To handle the "curse of dimensionality" introduced by such a massive feature vector, the authors employ regularized Logistic Regression. They explore three flavors of sparsity:

  • LR1 ($\ell_1$): Standard sparsity, which picks individual dimensions.
  • LRG (Group Lasso): "Object sparsity," which selects entire objects across all scales and locations.
  • LRG1 (Sparse Group Lasso): Jointly enforces both object-level and feature-level sparsity.

Because the features represent actual objects, this dimensionality reduction acts as "semantic compression." The representation is remarkably robust; the authors found they could discard 99% of the feature dimensions and still maintain over 70% classification accuracy. The model effectively learns which objects are irrelevant to a specific scene category and ignores them.

Key Results and Insights

The Object Bank model was evaluated on four major scene benchmarks: 15-Scene, a 9-class subset of LabelMe, UIUC-Sports, and the highly challenging MIT Indoor Scene dataset.

By treating objects as visual words, the OB representation significantly outperformed existing state-of-the-art methods that relied on low-level features:

  • MIT Indoor: Achieved 37.6% accuracy, a massive jump over the previous best of 26%. This proves that for cluttered indoor environments (like distinguishing a bedroom from a living room), the specific arrangement of objects is exactly what defines the category.
  • UIUC-Sports: Achieved 76.3%, outperforming the previous best of 73.4%.
  • 15-Scene: Achieved 80.9%, performing on par with the best existing specialized models.

To validate that the semantic meaning was the true driver of success, the authors tested a "Pseudo-Object Bank" using randomly generated detectors. It performed significantly worse, confirming that the semantic nature of the detectors is crucial.

Limitations and Future Directions

While powerful, the Object Bank approach has limitations. Running 200 different detectors at 12 scales is computationally expensive compared to calculating simple gradients. Furthermore, the system's performance is strictly capped by the accuracy of the underlying object detectors; if a detector for a crucial object fails, scene classification suffers.

However, the framework is agnostic to the specific detectors used. As object detection technology improves, the Object Bank representation automatically becomes more powerful. The authors envision scaling this approach to thousands or millions of object detectors as large-scale datasets continue to grow, potentially creating a truly universal visual vocabulary for machines.

Summary updated

Abstract

Robust low-level image features have been proven to be effective representations for a variety of visual recognition tasks such as object recognition and scene classification; but pixels, or even local image patches, carry little semantic meanings. For high level visual tasks, such low-level image representations are potentially not enough. In this paper, we propose a high-level image representation, called the Object Bank, where an image is represented as a scale invariant response map of a large number of pre-trained generic object detectors, blind to the testing dataset or visual task. Leveraging on the Object Bank representation, superior performances on high level visual recognition tasks can be achieved with simple off-the-shelf classifiers such as logistic regression and linear SVM. Sparsity algorithms make our representation more efficient and scalable for large scene datasets, and reveal semantically meaningful feature patterns.
BibTeX
@inproceedings{li2010object,
  author = {Li-Jia Li and Hao Su and Eric P. Xing and L. Fei-Fei},
  title = {{Object Bank: A High-Level Image Representation for Scene Classification \& Semantic Feature Sparsification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2010},
  url = {https://openreview.net/forum?id=r1Zp-wWdbH},
}
Figure