DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling

Akis Linardos, Matthias Kümmerer, Ori Press, Matthias Bethge

DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling

Introduction

Saliency prediction—the task of predicting where a human will look when viewing an image—is a cornerstone of computer vision. Since 2014, progress in this field has been driven almost entirely by transfer learning from ImageNet-trained models. However, performance had recently plateaued. DeepGaze IIE breaks this multi-year stagnation by ensembling diverse ImageNet backbones to better mimic human visual attention.

The authors identify two major hurdles in current research:

  1. Diminishing Returns: Simply using a "better" ImageNet classifier (e.g., moving from ResNet50 to EfficientNetB5) no longer yields automatic improvements in saliency prediction.
  2. Overconfidence: While individual models perform well on familiar datasets, they are often "overconfident" on new, out-of-domain images, predicting fixations with high certainty in places where humans don't actually look.

Methods

The researchers built upon the DeepGaze II architecture but introduced a more principled ensembling approach.

The DeepGaze IIE architecture (a) uses a fixed CNN backbone and a readout network. The ensemble approach (b) combines multiple diverse backbones to improve robustness and calibration.
  • The Architecture: They use a fixed (non-finetuned) CNN backbone to extract features, followed by a trainable "readout network" consisting of $1 \times 1$ convolutions. This setup is followed by a smoothing blur, a center-bias prior (accounting for the human tendency to look at the middle of images), and a softmax to produce a probability distribution of fixations.
  • Inter-Model Complementarity: Instead of picking one "best" backbone, they combined four diverse architectures: ShapeNet-C, EfficientNet-B5, ResNext-50, and DenseNet-201. They found that these models make different types of errors, and averaging their predictions cancels out individual biases.
  • Intra-Model Complementarity: They further improved robustness by averaging multiple instances of the same backbone trained with different random seeds.

Key Insights

A critical finding of this work is that ImageNet classification accuracy is no longer a reliable proxy for saliency performance. Once a backbone reaches a certain level of complexity, its performance on saliency tasks plateaus or even fluctuates regardless of further gains in classification accuracy.

Performance comparison across four datasets (MIT1003, SALICON, PASCAL-S, Toronto). The plots show that while individual backbones plateau, the DeepGaze IIE ensemble significantly outperforms them and approaches the human gold standard.

Furthermore, individual models are consistently overconfident on unseen data. However, the ensemble model (DeepGaze IIE) is remarkably well-calibrated. It "knows what it doesn't know," making it much more reliable for real-world applications. This success suggests that different CNN architectures encode fundamentally different types of visual information relevant to human attention, even if they achieve similar scores on classification tasks.

Key Results

  • New State-of-the-Art: DeepGaze IIE claimed the top spot on the MIT/Tuebingen Saliency Benchmark across all major metrics, including AUC (88.3%), sAUC (79.4%), and CC (82.4%).
  • Closing the Gap: On the MIT1003 dataset, the model achieved a 15 percentage point improvement over its predecessor (DeepGaze II), reaching 93% of the "gold standard" (the theoretical limit of human-to-human consistency).
  • Superior Generalization: While individual models saw performance drops of up to 20% on out-of-domain datasets like PASCAL-S, the ensemble remained robust and well-calibrated.

Limitations and Future Work

The authors noted diminishing returns in ensembling; adding a fifth backbone (ResNet50) actually decreased performance, suggesting a limit to beneficial averaging. Future work may explore backbones trained via self-supervised learning or on different tasks like object detection to provide even more diverse perspectives. Additionally, the computational cost of ensembling four large models remains a challenge for real-time or mobile applications.

Summary updated

Abstract

Since 2014 transfer learning has become the key driver for the improvement of spatial saliency prediction; however, with stagnant progress in the last 3-5 years. We conduct a large-scale transfer learning study which tests different ImageNet backbones, always using the same read out architecture and learning protocol adopted from DeepGaze II. By replacing the VGG19 backbone of DeepGaze II with ResNet50 features we improve the performance on saliency prediction from 78% to 85%. However, as we continue to test better ImageNet models as backbones (such as EfficientNetB5) we observe no additional improvement on saliency prediction. By analyzing the backbones further, we find that generalization to other datasets differs substantially, with models being consistently overconfident in their fixation predictions. We show that by combining multiple backbones in a principled manner a good confidence calibration on unseen datasets can be achieved. This new model, "DeepGaze IIE", yields a significant leap in benchmark performance in and out-of-domain with a 15 percent point improvement over DeepGaze II to 93% on MIT1003, marking a new state of the art on the MIT/Tuebingen Saliency Benchmark in all available metrics (AUC: 88.3%, sAUC: 79.4%, CC: 82.4%).
BibTeX
@inproceedings{linardos2021deepgaze,
  author = {Akis Linardos and Matthias Kümmerer and Ori Press and Matthias Bethge},
  title = {{DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01268},
}
Figure