Selective visual attention enables learning and recognition of multiple objects in cluttered scenes

Dirk B. Walther, Ueli Rutishauser, Christof Koch, Pietro PeronaCVIU 2005 · DOI

TL;DR

This paper introduces a novel approach to object recognition in cluttered scenes by integrating a bottom-up visual attention mechanism. By focusing computational resources on salient regions, the system significantly improves the ability to learn and recognize multiple objects from single images, even when they are partially occluded or surrounded by distracting elements.

What Problem Does This Paper Address?

Recognizing objects in real-world images is a significant challenge, especially when scenes contain multiple objects, clutter, or partial occlusions. Traditional object recognition methods often struggle because they require segmented and labeled objects for training, or they assume the target object is the dominant feature in an image. This makes it difficult to train systems on unlabeled images with complex scenes. The core problem is how to efficiently and reliably identify and learn individual objects within such challenging visual environments, mimicking the human ability to focus on relevant items and ignore distractions.

Methods

The authors propose a method that combines a bottom-up saliency-based visual attention system with Lowe's SIFT (Scale-Invariant Feature Transform) algorithm for object recognition.

  • Bottom-up Saliency-Based Attention: The attention system, based on Itti et al.'s model, computes "conspicuity maps" for various features (color, intensity, orientation) across different scales. These maps are combined into a single "saliency map," and a winner-take-all network identifies the most salient location.
  • Region Estimation: A key novelty is the extension of the saliency model to estimate the extent of the salient region likely to contain an object. This is achieved by tracing back the most influential feature map at the salient location and segmenting the region using adaptive thresholding and region growing.
  • Contrast Modulation: The image is then modulated by a mask derived from the estimated salient region, effectively enhancing contrast within the attended area and suppressing it elsewhere. This focuses the subsequent processing on potentially object-rich areas.
  • Object Recognition (SIFT): Lowe's SIFT algorithm is used for feature extraction and matching. By applying SIFT only to the contrast-modulated, attended regions, the system benefits from reduced clutter and a more focused set of keypoints.
  • Learning and Recognition Process: The system operates in a loop: it selects a salient region, extracts SIFT keypoints, and attempts to match them against learned object models. If a match is found, the model's counter is incremented; otherwise, a new model is learned from the attended region. This process is repeated for multiple salient regions (N fixations) to learn and recognize multiple objects.

Datasets

  • Robot-collected Video Sequence: A dataset of 1749 images collected by a robot navigating an indoor environment was used. This dataset features repeated objects and natural clutter.
  • Synthetically Generated Images: For experiments on cluttered scenes, the authors generated artificial images by merging 21 distinct objects with natural background images at various "relative object sizes" (ROS), controlling the amount of clutter.

Key Insights and Ideas

  • Attention as a Pre-processor for Recognition: The paper demonstrates that visual attention is not just for enhancing perception but can serve as a crucial pre-processing step for object recognition, especially in complex scenes.
  • Object Region Estimation is Key: Simply identifying a salient point is insufficient. Estimating the extent of the salient region is vital for effectively isolating potential objects from clutter.
  • Focusing SIFT is More Robust: Applying SIFT keypoint extraction only within attended regions significantly reduces the impact of background clutter and improves the reliability of object learning and recognition.
  • Bottom-up Attention is Sufficient for Basic Clutter Handling: The experiments show that even without top-down knowledge, bottom-up saliency can effectively guide attention to object-like regions and improve performance in cluttered environments.

Key Results

  • Improved Recognition in Clutter: In the robot video experiment, saliency-based region selection yielded 32 false positives (0.8%) compared to 81 (6.8%) for random region selection. "Good" object patches (recognized at least 10 times) were 87 with attention vs. 14 with random selection.
  • Efficient Learning of Multiple Objects: In natural scenes, the system learned models for two objects (a book and a box) from a single training image and recognized them with high accuracy (91% for the box, 58% for the book) in test images, with very few false positives.
  • Sustained Performance Improvement with Clutter: In synthetic cluttered scenes, attention consistently increased the true positive rate across various levels of clutter (ROS). In the most challenging case (0.05% ROS), attention boosted the true positive rate by a factor of 10.
  • Computational Efficiency: The attention mechanism helps by limiting processing to attended regions, saving computational resources compared to processing the entire image.

Limitations and Future Work

  • Bottom-up Focus: The current work relies solely on bottom-up attention. The authors acknowledge that top-down knowledge (e.g., prior knowledge about scenes or objects) can be crucial for cases where behaviorally relevant objects are not inherently salient.
  • Object-Based Attention: The current attention is primarily location-based. Future work could explore object-based attention, where attention is guided by object identity, potentially leading to iterative refinement of attention regions.
  • Scale and Ambiguity: Questions remain about how the system handles objects of vastly different scales or multiple objects within a single attended region. The authors suggest that feedback loops between the recognition and attention systems could address these issues.
  • Integration with Recognition: While attention significantly helps, the performance is still limited by the recognition system itself. Future work could explore how attention and recognition can be more tightly integrated.

Summary updated

BibTeX
@article{walther2005selective,
  author = {Dirk B. Walther and Ueli Rutishauser and Christof Koch and Pietro Perona},
  title = {{Selective visual attention enables learning and recognition of multiple objects in cluttered scenes}},
  journal = {Comput. Vis. Image Underst.},
  year = {2005},
  doi = {10.1016/j.cviu.2004.09.004},
}
Figure