TL;DR
This paper introduces DeepGaze IIE, a novel saliency prediction model that significantly improves performance by ensembling diverse ImageNet backbones. By addressing overconfidence issues in individual models, DeepGaze IIE achieves state-of-the-art results on unseen datasets, demonstrating the power of principled ensemble learning for robust visual attention modeling.
What Problem Does This Paper Address?
Predicting where humans will look on an image (visual saliency) is a fundamental problem in computer vision with applications ranging from image compression to user interface design. While transfer learning from large image classification datasets like ImageNet has been the primary driver of progress in saliency prediction since 2014, progress has plateaued in recent years. Existing models, even those using powerful ImageNet backbones, often struggle with generalization to new datasets and can be overconfident in their predictions, meaning they are too certain about where people will look, even when they are wrong. This lack of robust calibration makes them less reliable in real-world scenarios where the data distribution might differ from the training set.
Methods
The core of this work lies in a systematic study of how different ImageNet-trained convolutional neural networks (CNNs) as "backbones" influence saliency prediction performance. The authors adopted the DeepGaze II architecture, which uses a fixed backbone and a trainable "readout" network (composed of 1x1 convolutions) to predict fixation densities.
Key methodological contributions include:
- Extensive Backbone Exploration: They tested a wide range of state-of-the-art ImageNet backbones (e.g., ResNet, EfficientNet, DenseNet) to understand their transferability to saliency.
- Principled Ensemble Learning: Instead of relying on a single best backbone, they developed an ensemble approach. This involved combining predictions from multiple diverse backbones, as well as multiple instances of the same backbone (intra-model complementarity), to leverage their complementary strengths.
- Novel Confidence Calibration Testing: Recognizing that standard calibration metrics are ill-suited for the high entropy of fixation prediction, they proposed a new method to assess how well a model's predicted probabilities align with actual fixation counts across different probability quantiles.
Datasets
- SALICON: Used for pre-training the models.
- MIT1003: The primary dataset used for training and evaluating the final DeepGaze IIE model. It consists of natural images with human fixation data.
- MIT300: Used for evaluating performance on unseen data.
- PASCAL-S and Toronto: Used for out-of-domain generalization testing.
Key Insights and Ideas
- ImageNet Performance Doesn't Directly Translate: While better ImageNet classification performance generally leads to better saliency prediction, this relationship plateaus. Simply using the "best" ImageNet classifier as a backbone doesn't guarantee the best saliency model.
- Generalization Varies Wildly: Different backbones exhibit significantly different generalization capabilities to unseen datasets. Some models are robust, while others perform poorly.
- Overconfidence is a Major Issue: Individual models are often highly overconfident in their predictions, especially when tested on out-of-domain data. This means they assign high probabilities to regions that don't receive many fixations.
- Ensembling is Key to Robustness and Calibration: Combining predictions from diverse backbones (inter-model complementarity) and even different instances of the same backbone (intra-model complementarity) significantly boosts performance and, crucially, improves confidence calibration, making the models more trustworthy on new data.
- 1x1 Convolutions are Powerful for Transfer Learning: The simple readout network of 1x1 convolutions, when combined with fixed deep features, proves to be an effective and interpretable way to adapt ImageNet features for saliency.
Key Results
- State-of-the-Art Performance: DeepGaze IIE achieved a significant leap in performance, reaching 93% on the MIT1003 benchmark, a 15 percentage point improvement over the previous state-of-the-art (DeepGaze II at 78%).
- Improved Generalization: The ensemble model showed much better performance on out-of-domain datasets like PASCAL-S and Toronto, achieving near gold-standard performance, while individual models struggled.
- Confidence Calibration: While individual models were overconfident on unseen data, DeepGaze IIE demonstrated near-perfect calibration on these datasets, making it more reliable.
- Backbone Performance Trends: Performance generally increased with backbone complexity up to ResNet-50/DenseNet-201, after which further improvements plateaued.
Limitations and Future Work
- Diminishing Returns of ImageNet Features: The paper suggests that ImageNet features may be reaching a point of diminishing returns for saliency prediction.
- Future Work: The authors propose exploring maximally diverse backbones, including those trained with self-supervised learning or for other tasks like object detection, to further improve performance. They also suggest that their principled ensemble learning approach could be applied to other domains.
Summary updated