TL;DR
This paper tackles the "black box" problem of deep saliency models by proposing a novel visualization technique to understand what these models learn. They demonstrate that deep saliency models learn high-level semantic information, not just low-level features, suggesting a closer alignment with human visual attention than previously thought.
What Problem Does This Paper Address?
Human visual attention is remarkably efficient, allowing us to process vast amounts of visual information by selectively focusing on important parts of a scene. Computational models of visual attention, particularly deep convolutional neural networks (CNNs), have achieved impressive performance in predicting where humans look (fixation prediction). However, these deep models are often treated as "black boxes" – we know they work well, but we don't fully understand why or how they learn to predict saliency. Existing methods for visualizing CNNs are often ill-suited for saliency models, which need to capture both salient and non-salient patterns. This paper aims to bridge this gap by providing a method to peer inside these deep saliency models and understand the features they learn.
Methods
The authors propose a two-pronged approach:
- A Simple Yet Effective Saliency Model: They introduce a straightforward, fully convolutional encoder-decoder architecture. This model uses a pre-trained VGG16 network for feature extraction and a novel residual decoder. Crucially, they propose a new pixel-wise loss function called Exponential Absolute Distance (EAD). This loss function is designed to provide a better gradient for small differences and a larger punishment for large errors compared to L1 or L2 losses, leading to improved performance.
- A Novel Visualization Method: To understand the learned representations, they adapt existing techniques. They use the Normalized Scan-path Saliency (NSS) score, a standard saliency evaluation metric, to identify neurons that are "positive fixation detectors" – meaning their activations correlate with human gaze. They then employ a network dissection approach (similar to Bau et al. [11]) to determine what objects or object parts these identified neurons are attuned to.
Datasets
The paper utilizes several established datasets for training and evaluation:
- SALICON [6]: A large-scale dataset for visual saliency, used for training and validation.
- MIT1003 [27]: Used for visualization and for comparing the performance of different loss functions.
- MIT300 [30]: A standard benchmark for human gaze prediction, used for comparing the proposed model against state-of-the-art methods.
- Broden [11]: A dataset with pixel-wise semantic labels, used for the network dissection part of the visualization method.
Key Insights and Ideas
- Deep Saliency Models Learn Semantic Information: The most significant finding is that high-performing deep saliency models learn to detect specific semantic categories (like animals, objects, and their parts) rather than just relying on low-level perceptual cues (like center-surround differences). This suggests that saliency, as perceived by humans, involves high-level world knowledge.
- Positive Fixation Detectors are Specific: Neurons identified as "positive fixation detectors" tend to be attuned to very specific visual patterns, often corresponding to unique objects or object parts (e.g., a face, a head).
- Negative Central Bias: Some neurons, even in models without an explicit central bias constraint, learn to inhibit the center of the image, suggesting this bias is learned from the training data.
- Simplicity Can Be Powerful: The proposed simple encoder-decoder architecture with the EAD loss function achieves performance comparable to or better than more complex, multi-architecture models.
Key Results
- The proposed EAD loss function outperforms other pixel-wise loss functions (L1, L2, BCE) on the MIT1003 dataset, achieving the best NSS, CC, AUC, and Sim scores.
- The proposed model performs on par or better than several state-of-the-art saliency models on the MIT300 dataset, particularly in terms of the NSS score.
- The visualization analysis reveals that positive fixation detectors in better-performing models are more numerous and highly correlated with human gaze.
- The network dissection shows that these detectors are attuned to semantic categories like animals (dogs, cats), objects (motorbikes, balls), and parts (head, hair).
Limitations and Future Work
- Model Complexity: While the authors propose a simple architecture, many state-of-the-art models are complex and difficult to interpret. The visualization method helps, but fully understanding all aspects of these complex models remains a challenge.
- Generalizability of Visualization: The visualization method is applied to specific models and datasets. Further work could explore its applicability to a wider range of architectures and visual domains.
- Closing the Gap: The authors believe their findings can help measure the gap between current saliency models and human inter-observer models, and they aim to build new models to close this gap. They plan to share their code to facilitate future research.
Summary updated