TL;DR
This paper introduces MENTOR, a novel pretraining method that leverages human saliency maps to improve the generalization capabilities of deep learning models, particularly in anomaly detection tasks with limited labeled data. By first training an autoencoder to learn human-like visual salience and then fine-tuning the encoder for a specific classification task, MENTOR significantly outperforms traditional pretraining methods and even existing human perception-guided approaches.
What Problem Does This Paper Address?
Deep learning models, especially Convolutional Neural Networks (CNNs), have achieved remarkable success in various computer vision tasks. However, their generalization ability, particularly in "open-set" recognition scenarios where new, unseen anomalies can appear, remains a challenge. This is especially true in domains like biometric presentation attack detection (PAD), synthetic face detection, and medical diagnostics, where acquiring large amounts of labeled data for every possible anomaly is difficult or impossible. While incorporating human perceptual information has shown promise in improving generalization, the optimal way to integrate this limited human data into model training is still an active research question. Existing methods often require complex architectural changes or struggle to balance human guidance with task-specific learning.
Methods
MENTOR proposes a two-stage pretraining approach:
- Step 1: Human Saliency Autoencoder: An autoencoder is trained to reconstruct human saliency maps from input images. Crucially, this step is performed without any class labels. The goal is for the encoder to learn to extract features that capture human visual attention and domain-specific salient regions, mimicking how humans learn to perceive the world before understanding specific object categories.
- Step 2: Domain-Specific Classifier Fine-tuning: The decoder of the autoencoder is removed, and a classification layer is added on top of the pre-trained encoder. This new model is then fine-tuned using standard cross-entropy loss and class labels for the specific anomaly detection task. Importantly, the encoder's weights are not frozen, allowing for further adaptation.
This disentangled approach allows the model to first build a strong, human-perception-informed feature representation before specializing it for a classification task.
Datasets
The paper evaluates MENTOR across three domains:
- Iris Presentation Attack Detection (PAD): Uses a dataset of bona fide and anomalous iris images with human saliency maps collected by Boyd et al. [3].
- Synthetic Face Detection: Utilizes real and synthetically generated face images with human saliency maps from [5].
- Chest X-ray-based Diagnosis: Employs normal and abnormal chest X-ray images with eye-tracking visual salience data from radiologists [2].
For evaluation, the paper uses test sets from the Iris Liveness Detection 2020 Competition (LivDet-2020) [11], synthesized images from various GAN architectures and authentic faces, and a disjoint set of chest X-ray images.
Key Insights and Ideas
- Human Perception as a Prior: The core idea is that human visual saliency provides a valuable "prior" for learning generalizable features, akin to how infants learn about the world by observing before being explicitly taught labels.
- Disentangled Pretraining: Separating the human saliency learning (Step 1) from the classification task (Step 2) allows the model to build robust, domain-agnostic perceptual features without being immediately constrained by class labels.
- Flexibility and Adaptability: MENTOR is architecture-agnostic and can be applied to various CNN backbones. It can also be used to initialize existing human perception-guided methods, further boosting their performance.
- Addressing Data Scarcity: The method is particularly effective in data-limited regimes where acquiring extensive labeled data is challenging, making it suitable for real-world anomaly detection scenarios.
Key Results
- Improved Generalization: MENTOR consistently outperforms models initialized with ImageNet weights and, in most cases, matches or surpasses state-of-the-art human perception-guided methods (CYBORG, UNET+Gaze) across all three domains (Iris PAD, Synthetic Face Detection, Chest X-ray).
- Strong Performance in Iris PAD: MENTOR achieved AUROC scores of up to 0.912±0.02 for Iris PAD with a ResNet152 backbone.
- Effective for Synthetic Face Detection: MENTOR significantly improved performance in synthetic face detection, where other saliency-based methods struggled to perform above random chance.
- Enhancing Other Methods: When used to initialize CYBORG and UNET+Gaze, MENTOR led to substantial gains in generalization performance, demonstrating its utility as a complementary technique.
- Qualitative Saliency Maps: MENTOR's generated saliency maps are visually comparable to human saliency maps and appear more focused and informative than those produced by other methods like UNET+Gaze and CYBORG, which sometimes suffer from information collapse or low resolution.
Limitations and Future Work
- Smaller Architectures: While MENTOR shows promise on smaller ResNet architectures, further investigation into optimal training configurations and optimizers for these models is suggested.
- Quantitative Saliency Analysis: The paper notes that while qualitative analysis of saliency maps is promising, more robust quantitative metrics are needed to fully assess the quality of generated saliency, especially given the sparsity of some human annotations.
- Combining with Other Methods: The authors suggest that combining MENTOR with other strong regularization loss components could further enhance generalization.
- Synthetic Saliency Generation: The paper briefly touches upon MENTOR's potential to generate synthetic saliency maps for use in other paradigms, which could be a fruitful area for future research.
Summary updated