TL;DR
This paper introduces a robust framework for predicting human eye fixations by using Multiple Kernel Learning (MKL) to intelligently fuse low-level visual cues with high-level semantic information. By incorporating ObjectBank—a massive library of 177 object detectors—the authors demonstrate that "intermediate integration" of features significantly outperforms traditional linear or ensemble-based saliency models.
Introduction: The Semantic Gap in Saliency
Visual saliency estimation aims to predict which parts of an image will grab a human's attention. While early models relied on simple bottom-up features (like color contrast or edge sharpness), modern computer vision has struggled with two main issues:
- Feature Integration: How do we combine different types of information (color, orientation, object presence) into a single "master" saliency map? Simple linear summation assumes every feature is equally important, which is rarely the case in human perception.
- The Semantic Gap: Humans don't just look at high-contrast pixels; we look at objects. Most existing models only accounted for a few "special" objects like faces or cars, failing to capture the full breadth of semantic context.
The authors at Hacettepe University tackle these challenges by treating saliency as a supervised learning problem, seeking a principled way to weight and combine a vast array of features.
Methodology
The core innovation is the use of Multiple Kernel Learning (MKL). In machine learning, kernels help find patterns in high-dimensional space. While a standard Support Vector Machine (SVM) uses a single kernel for all features (early integration), MKL allows different features to have their own kernels which are then combined.
The authors argue this provides "intermediate integration"—a sweet spot that avoids the over-simplification of early integration and the over-independence of late integration (like AdaBoost).
The Feature Pipeline
- High-Level (Semantic): The model utilizes ObjectBank, a filter bank of 177 different object detectors (e.g., "building," "car," "street lamp") applied at 5 scales, resulting in 885 feature dimensions.
- Mid-Level: A horizon line detector (since humans look for objects relative to the ground) and "Gist" features.
- Low-Level: Traditional cues like intensity, orientation, and color contrast (via the Itti-Koch model), plus subband features and color histograms.
- Priors: A "Center Map" to account for the human tendency to look at the middle of a photograph.
The authors tested three MKL variants: RBMKL (fixed-rule), NLMKL (non-linear), and LMKL (localized).
Datasets and Evaluation
The model was trained and evaluated on two gold-standard eye-tracking datasets:
- MIT1003: 1,003 images (landscapes and portraits) with fixation data from 15 viewers. This was used for the primary training and comparative analysis.
- Toronto: 120 natural images with data from 20 subjects, used to test the model's ability to generalize to new data.
Key Results and Insights
Performance was measured using the Area Under the Curve (AUC), where 1.0 is a perfect prediction and 0.5 is random chance.
- Superior Performance: The RBMKL model achieved a "Best" AUC of 0.841 on MIT1003, outperforming linear SVM (0.830) and AdaBoost (0.816). On the Toronto dataset, it achieved an AUC of 0.831, significantly higher than the classical Itti et al. bottom-up model (0.741).
- Objects Drive Attention: A model using only object-level features performed nearly as well as the classic Itti-Koch bottom-up model, confirming that semantic context is a primary driver of human gaze.
- Non-linear Relationships: The success of the RBMKL variant (which uses kernel multiplication) suggests that visual features "amplify" each other non-linearly rather than just adding up.
Conclusion
By learning the weights of different kernels, the MKL model automatically discovers which features are most relevant for specific types of scenes. While the computational cost of running 885 object detectors remains a hurdle for real-time applications, this framework successfully bridges the gap between low-level pixels and high-level semantic understanding in visual saliency.
Summary updated