Using the forest to see the trees: exploiting context for visual object detection and localization

Antonio Torralba, Kevin P. Murphy, William T. Freeman

TL;DR

This paper tackles the challenge of robust object detection by integrating "global" scene context with traditional "local" image features. By understanding the overall scene (e.g., a street vs. a forest), the system can predict where objects are likely to appear and how many, significantly reducing false positives and improving detection accuracy.

What Problem Does This Paper Address?

Recognizing objects in images is a fundamental problem in computer vision, with applications ranging from autonomous driving to image retrieval. Traditional methods often rely on exhaustive "sliding window" approaches, where an object detector is applied to every possible location and scale in an image. While effective for certain tasks, this brute-force method suffers from two major drawbacks: it's computationally expensive and prone to generating many false positives, especially when dealing with numerous object categories. This is because these methods largely ignore the rich contextual information present in real-world scenes, where objects rarely appear in isolation but rather co-occur in predictable ways. For instance, a car is highly likely to be found on a street, but very unlikely on a beach.

Methods

The authors propose an integrated probabilistic framework that combines two sources of information:

  • Local Features (Bottom-Up): Standard object detectors that analyze small image patches for specific features (e.g., Gabor filter responses) to identify potential objects. These are represented by a confidence score ($c$) for each detection.
  • Global Features (Top-Down): A "gist" descriptor, which captures the overall spatial layout and texture of the scene without explicitly identifying objects. This global context is used to:
  • Scene Classification: Predict the type of scene (e.g., street, highway, indoor).
  • Location Priming: Predict the likely vertical location of objects within the scene.
  • Object Counting: Predict the probable number of objects of a given class present in the scene.

The core innovation lies in how these two information streams are fused. Instead of treating detections independently, the system uses the global scene information to modulate the confidence of local detections. For example, if the scene is classified as a "beach," detections of "cars" in unlikely locations or in large numbers will have their confidence scores significantly reduced. Conversely, a confident detection in a contextually plausible location is boosted. This is achieved through a probabilistic graphical model that explicitly encodes the relationships between scene category, object presence, count, location, and detector scores.

Datasets

The paper primarily uses the scenes dataset from Oliva and Torralba [11], which contains 2688 images across 8 scene categories. They also mention using a subset of the LabelMe dataset [15] for training and testing object detectors.

Key Insights and Ideas

  • Context is King (for reducing false positives): The most significant insight is that understanding the overall scene context dramatically reduces false positives in object detection. Even if a local detector is fooled, the scene context can often correct it.
  • "Gist" as a Powerful Scene Descriptor: The "gist" descriptor, a compact representation of global image features, is surprisingly effective at capturing scene semantics and predicting object presence and location.
  • Hierarchical Approach to Context: Instead of relying on detecting other objects to infer context (a chicken-and-egg problem), the paper proposes using the scene category as an intermediate, more easily inferred context.
  • Probabilistic Fusion for Robustness: The integration of local and global information is framed within a probabilistic graphical model, allowing for principled fusion and uncertainty management.
  • Location Priming: Using global features to predict the likely vertical location of objects helps to suppress detections in implausible vertical positions.

Key Results

  • The integrated system significantly outperforms object detectors relying solely on local features, particularly in terms of precision (reducing false positives).
  • While the global context doesn't increase the recall (i.e., it doesn't help find more objects that the local detector missed), it substantially improves precision by eliminating detections in contextually inappropriate scenes.
  • The scene classification component, based on the gist descriptor, achieves approximately 75% accuracy across 15 scene categories.
  • The paper demonstrates qualitative improvements in Figures 6 and 7, showing how context effectively suppresses false detections and refines object locations.

Limitations and Future Work

  • Contextually Plausible False Positives: The system can still be fooled if a false positive detection occurs in a contextually plausible location (e.g., a car detected high in the sky in a street scene).
  • Ambiguous Local Evidence: If the local detector provides very weak or ambiguous evidence, the system might still eliminate a correct detection if it's slightly out of the predicted context.
  • Finer-Grained Context: The authors suggest that a more detailed notion of context, perhaps based on the co-occurrence of specific objects, could further improve performance.
  • Image Retrieval Applications: For applications like web image retrieval, object presence detection (without precise localization) is often sufficient and can be sped up by using gist-based filtering.
  • Future work could explore more sophisticated context models and further integration of local and global cues.

Summary updated

Abstract

Recognizing objects in images is an active area of research in computer vision. In the last two decades, there has been much progress and there are already object recognition systems operating in commercial products. However, most of the algorithms for detecting objects perform an exhaustive search across all locations and scales in the image comparing local image regions with an object model. That approach ignores the semantic structure of scenes and tries to solve the recognition problem by brute force. In the real world, objects tend to covary with other objects, providing a rich collection of contextual associations. These contextual associations can be used to reduce the search space by looking only in places in which the object is expected to be; this also increases performance, by rejecting patterns that look like the target but appear in unlikely places. Most modeling attempts so far have defined the context of an object in terms of other previously recognized objects. The drawback of this approach is that inferring the context becomes as difficult as detecting each object. An alternative view of context relies on using the entire scene information holistically. This approach is algorithmically attractive since it dispenses with the need for a prior step of individual object recognition. In this paper, we use a probabilistic framework for encoding the relationships between context and object properties and we show how an integrated system provides improved performance. We view this as a significant step toward general purpose machine vision systems.
BibTeX
@article{torralba2010using,
  author = {Antonio Torralba and Kevin P. Murphy and William T. Freeman},
  title = {{Using the forest to see the trees: exploiting context for visual object detection and localization}},
  journal = {Communications of the ACM},
  year = {2010},
  doi = {10.1145/1666420.1666446},
}
Figure