Visual saliency estimation by integrating features using multiple kernel learning

Yasin Kavak, Erkut Erdem, Aykut ErdemarXiv 2013 · arXiv

TL;DR

This paper introduces a robust framework for predicting human eye fixations by using Multiple Kernel Learning (MKL) to intelligently fuse low-level visual cues with high-level semantic information. By incorporating ObjectBank—a massive library of 177 object detectors—the authors demonstrate that "intermediate integration" of features significantly outperforms traditional linear or ensemble-based saliency models.

Introduction: The Semantic Gap in Saliency

Visual saliency estimation aims to predict which parts of an image will grab a human's attention. While early models relied on simple bottom-up features (like color contrast or edge sharpness), modern computer vision has struggled with two main issues:

  1. Feature Integration: How do we combine different types of information (color, orientation, object presence) into a single "master" saliency map? Simple linear summation assumes every feature is equally important, which is rarely the case in human perception.
  2. The Semantic Gap: Humans don't just look at high-contrast pixels; we look at objects. Most existing models only accounted for a few "special" objects like faces or cars, failing to capture the full breadth of semantic context.

The authors at Hacettepe University tackle these challenges by treating saliency as a supervised learning problem, seeking a principled way to weight and combine a vast array of features.

Methodology

The core innovation is the use of Multiple Kernel Learning (MKL). In machine learning, kernels help find patterns in high-dimensional space. While a standard Support Vector Machine (SVM) uses a single kernel for all features (early integration), MKL allows different features to have their own kernels which are then combined.

Overview of the proposed saliency estimation framework, illustrating the integration of high-level object features with low-level visual cues via Multiple Kernel Learning.

The authors argue this provides "intermediate integration"—a sweet spot that avoids the over-simplification of early integration and the over-independence of late integration (like AdaBoost).

The Feature Pipeline

  • High-Level (Semantic): The model utilizes ObjectBank, a filter bank of 177 different object detectors (e.g., "building," "car," "street lamp") applied at 5 scales, resulting in 885 feature dimensions.
  • Mid-Level: A horizon line detector (since humans look for objects relative to the ground) and "Gist" features.
  • Low-Level: Traditional cues like intensity, orientation, and color contrast (via the Itti-Koch model), plus subband features and color histograms.
  • Priors: A "Center Map" to account for the human tendency to look at the middle of a photograph.

The authors tested three MKL variants: RBMKL (fixed-rule), NLMKL (non-linear), and LMKL (localized).

Datasets and Evaluation

The model was trained and evaluated on two gold-standard eye-tracking datasets:

  • MIT1003: 1,003 images (landscapes and portraits) with fixation data from 15 viewers. This was used for the primary training and comparative analysis.
  • Toronto: 120 natural images with data from 20 subjects, used to test the model's ability to generalize to new data.
Sample images from the evaluation datasets showing human fixation points (yellow circles) across diverse scenes.

Key Results and Insights

Performance was measured using the Area Under the Curve (AUC), where 1.0 is a perfect prediction and 0.5 is random chance.

  • Superior Performance: The RBMKL model achieved a "Best" AUC of 0.841 on MIT1003, outperforming linear SVM (0.830) and AdaBoost (0.816). On the Toronto dataset, it achieved an AUC of 0.831, significantly higher than the classical Itti et al. bottom-up model (0.741).
  • Objects Drive Attention: A model using only object-level features performed nearly as well as the classic Itti-Koch bottom-up model, confirming that semantic context is a primary driver of human gaze.
  • Non-linear Relationships: The success of the RBMKL variant (which uses kernel multiplication) suggests that visual features "amplify" each other non-linearly rather than just adding up.

Conclusion

By learning the weights of different kernels, the MKL model automatically discovers which features are most relevant for specific types of scenes. While the computational cost of running 885 object detectors remains a hurdle for real-time applications, this framework successfully bridges the gap between low-level pixels and high-level semantic understanding in visual saliency.

Summary updated

Abstract

In the last few decades, significant achievements have been attained in predicting where humans look at images through different computational models. However, how to determine contributions of different visual features to overall saliency still remains an open problem. To overcome this issue, a recent class of models formulates saliency estimation as a supervised learning problem and accordingly apply machine learning techniques. In this paper, we also address this challenging problem and propose to use multiple kernel learning (MKL) to combine information coming from different feature dimensions and to perform integration at an intermediate level. Besides, we suggest to use responses of a recently proposed filterbank of object detectors, known as Object-Bank, as additional semantic high-level features. Here we show that our MKL-based framework together with the proposed object-specific features provide state-of-the-art performance as compared to SVM or AdaBoost-based saliency models.
BibTeX
@misc{kavak2013visual,
  author = {Yasin Kavak and Erkut Erdem and Aykut Erdem},
  title = {{Visual saliency estimation by integrating features using multiple kernel learning}},
  year = {2013},
  eprint = {1307.5693},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1307.5693},
}
Figure