DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling
Introduction
Saliency prediction—the task of predicting where a human will look when viewing an image—is a cornerstone of computer vision. Since 2014, progress in this field has been driven almost entirely by transfer learning from ImageNet-trained models. However, performance had recently plateaued. DeepGaze IIE breaks this multi-year stagnation by ensembling diverse ImageNet backbones to better mimic human visual attention.
The authors identify two major hurdles in current research:
- Diminishing Returns: Simply using a "better" ImageNet classifier (e.g., moving from ResNet50 to EfficientNetB5) no longer yields automatic improvements in saliency prediction.
- Overconfidence: While individual models perform well on familiar datasets, they are often "overconfident" on new, out-of-domain images, predicting fixations with high certainty in places where humans don't actually look.
Methods
The researchers built upon the DeepGaze II architecture but introduced a more principled ensembling approach.
- The Architecture: They use a fixed (non-finetuned) CNN backbone to extract features, followed by a trainable "readout network" consisting of $1 \times 1$ convolutions. This setup is followed by a smoothing blur, a center-bias prior (accounting for the human tendency to look at the middle of images), and a softmax to produce a probability distribution of fixations.
- Inter-Model Complementarity: Instead of picking one "best" backbone, they combined four diverse architectures: ShapeNet-C, EfficientNet-B5, ResNext-50, and DenseNet-201. They found that these models make different types of errors, and averaging their predictions cancels out individual biases.
- Intra-Model Complementarity: They further improved robustness by averaging multiple instances of the same backbone trained with different random seeds.
Key Insights
A critical finding of this work is that ImageNet classification accuracy is no longer a reliable proxy for saliency performance. Once a backbone reaches a certain level of complexity, its performance on saliency tasks plateaus or even fluctuates regardless of further gains in classification accuracy.
Furthermore, individual models are consistently overconfident on unseen data. However, the ensemble model (DeepGaze IIE) is remarkably well-calibrated. It "knows what it doesn't know," making it much more reliable for real-world applications. This success suggests that different CNN architectures encode fundamentally different types of visual information relevant to human attention, even if they achieve similar scores on classification tasks.
Key Results
- New State-of-the-Art: DeepGaze IIE claimed the top spot on the MIT/Tuebingen Saliency Benchmark across all major metrics, including AUC (88.3%), sAUC (79.4%), and CC (82.4%).
- Closing the Gap: On the MIT1003 dataset, the model achieved a 15 percentage point improvement over its predecessor (DeepGaze II), reaching 93% of the "gold standard" (the theoretical limit of human-to-human consistency).
- Superior Generalization: While individual models saw performance drops of up to 20% on out-of-domain datasets like PASCAL-S, the ensemble remained robust and well-calibrated.
Limitations and Future Work
The authors noted diminishing returns in ensembling; adding a fifth backbone (ResNet50) actually decreased performance, suggesting a limit to beneficial averaging. Future work may explore backbones trained via self-supervised learning or on different tasks like object detection to provide even more diverse perspectives. Additionally, the computational cost of ensembling four large models remains a challenge for real-time or mobile applications.
Summary updated