Visual saliency estimation by nonlinearly integrating features using region covariances

Erkut Erdem, Aykut ErdemJournal of Vision 2013 · DOI

TL;DR

This paper introduces a novel approach to visual saliency estimation by using region covariance descriptors to nonlinearly integrate image features. This method significantly outperforms existing models in predicting human eye fixations, detecting salient objects, and image retargeting, by better capturing local image structures and their correlations.

What Problem Does This Paper Address?

Our visual system is remarkably adept at quickly identifying the most important parts of a scene, a process known as visual saliency. This is crucial for managing the overwhelming amount of visual information we constantly receive. Computational models aim to replicate this by identifying "salient" regions. Traditional bottom-up saliency models typically extract features like color and orientation independently, create separate "maps" for each, and then combine them linearly. However, this linear combination often fails to capture the complex, non-linear interactions between features that are essential for accurate saliency prediction. The authors address this gap by proposing a method that inherently models these feature correlations, leading to a more nuanced understanding of what makes a region visually salient.

Methods

The core innovation of this paper lies in the use of region covariance descriptors as "meta-features" for saliency estimation. Instead of processing individual feature maps linearly, the authors represent local image patches (regions) by their covariance matrices. These matrices capture not only the variance of individual features (like color or orientation) but also their correlations with each other. This naturally leads to a nonlinear integration of features, as the covariance matrix implicitly models how different features co-vary within a region.

The saliency of a region is then computed by measuring the dissimilarity between its covariance descriptor and those of its surrounding regions. This is done using a metric based on generalized eigenvalues and eigenvectors of the covariance matrices. The authors also propose an extension to incorporate first-order statistics (mean), which can further improve performance, especially in cases where simple intensity differences are key to saliency.

Datasets

The proposed models were evaluated on several publicly available datasets:

  • Toronto data set: 120 natural color images used for eye fixation prediction.
  • MIT1003 data set: A large dataset of 1003 natural color images (landscape and portrait) for eye fixation prediction.
  • MIT300 data set: 300 natural images for eye fixation prediction.
  • ImgSal data set: 235 natural color images with ground truth for salient object detection, categorized by difficulty.
  • ReTargetMe data set: 80 images with 92 resizing scenarios used for evaluating image retargeting performance.

Key Insights and Ideas

  • Covariance as a Nonlinear Integrator: The most significant insight is that region covariance matrices inherently provide a powerful way to nonlinearly integrate multiple visual features by modeling their correlations. This moves beyond simple linear summation and better reflects how complex visual information is processed.
  • Local Structure Matters: Region covariances capture local image structures more effectively than standard linear filters, which is crucial for understanding saliency in textured or patterned natural scenes.
  • First-Order Statistics Complement Second-Order: While region covariances excel at capturing feature interactions, incorporating simple mean values (first-order statistics) can further enhance saliency prediction, particularly for regions with distinct intensity differences.
  • Biological Plausibility: The authors speculate that this approach might align with biological mechanisms, citing research suggesting that complex cells in the visual cortex might encode statistical variations through covariance-like processes.

Key Results

The proposed region covariance-based saliency models demonstrated superior performance across various tasks:

  • Eye Fixation Prediction: The models consistently outperformed state-of-the-art methods on the Toronto, MIT1003, and MIT300 datasets, achieving higher AUC and NSS scores and lower EMD and similarity scores. The "covariances + means + center" model was particularly strong.
  • Salient Object Detection: On the ImgSal dataset, the proposed models outperformed others in several categories, especially those involving complex textures and repeating distractors.
  • Image Retargeting: The models showed strong performance on the ReTargetMe dataset, particularly in preserving prominent objects and structures during resizing, outperforming most other methods in EMD scores.
  • Psychophysical Patterns: The models successfully reproduced pop-out phenomena for various stimuli, demonstrating their ability to capture basic visual search mechanisms.

Limitations and Future Work

  • Spatial Coincidence Assumption: The authors acknowledge a limitation in their multiscale approach, which relies on a "spatial coincidence assumption." This might make it challenging to detect multiple salient objects that appear at different scales. They suggest extending the framework to work simultaneously across scales.
  • Incorporating Top-Down Information: While the current model is bottom-up, the authors suggest exploring the incorporation of more complex semantic features (like scene gist, faces, or text) to integrate task-oriented top-down influences.
  • Dynamic Scenes: Future work could explore extending the model to estimate spatio-temporal saliency in dynamic scenes.
  • Computational Efficiency: While the model is reasonably efficient, further improvements could be made by using MEX C++ subroutines or parallelizing the code.

Summary updated

Abstract

Abstract To detect visually salient elements of complex natural scenes, computational bottom-up saliency models commonly examine several feature channels such as color and orientation in parallel. They compute a separate feature map for each channel and then linearly combine these maps to produce a master saliency map. However, only a few studies have investigated how different feature dimensions contribute to the overall visual saliency. We address this integration issue and propose to use covariance matrices of simple image features (known as region covariance descriptors in the computer vision community; Tuzel, Porikli, & Meer, 2006) as meta-features for saliency estimation. As low-dimensional representations of image patches, region covariances capture local image structures better than standard linear filters, but more importantly, they naturally provide nonlinear integration of different features by modeling their correlations. We also show that first-order statistics of features could be easily incorporated to the proposed approach to improve the performance. Our experimental evaluation on several benchmark data sets demonstrate that the proposed approach outperforms the state-of-art models on various tasks including prediction of human eye fixations, salient object detection, and image-retargeting.
BibTeX
@article{erdem2013visual,
  author = {Erkut Erdem and Aykut Erdem},
  title = {{Visual saliency estimation by nonlinearly integrating features using region covariances}},
  journal = {Journal of Vision},
  year = {2013},
  doi = {10.1167/13.4.11},
}
Figure