Abstract
Attention mechanisms are pivotal in modern deep learning, yet they often lack explicit supervision, leading to reliance on irrelevant background features. This paper presents a novel self-supervised approach to refine attention maps without human-annotated labels. By generating "surrogate" ground-truth attention maps through input masking and confidence probing, the proposed method distills discriminative patterns back into the network. Experimental results across vision, text, and multi-modal tasks demonstrate significant improvements in both predictive accuracy and alignment with human attention.
Introduction
While attention modules allow models to focus on salient input regions, they are typically trained implicitly. In tasks like Visual Question Answering (VQA), a model might correctly answer a question while attending to irrelevant background noise rather than the target object. This lack of grounding reduces interpretability and robustness. Although human-annotated datasets like VQA-HAT provide explicit supervision, they are prohibitively expensive to scale. The authors address this by asking: can we provide supervision to an attention module without manual labeling?
Methodology
The proposed framework introduces a two-stage pipeline for self-supervised attention.
1. Generating Surrogate Attention Maps
To identify truly discriminative regions, the authors probe a pre-trained baseline model using randomly masked versions of the input. For images, 500 random masks are applied; for text, two words are masked at a time. The model's confidence in the ground-truth class is recorded for each variation. The mask that preserves the highest confidence is designated as the "surrogate ground truth" (Att-hat). This process assumes that the regions visible in the highest-confidence mask are the most critical for the task.
2. Training with Attention Supervision
The network is then re-trained using a multi-task loss function combining standard classification loss with an attention loss. The attention loss penalizes the discrepancy between the model’s internal attention and the surrogate map. The authors found that an adversarial loss—utilizing a discriminator to align the distributions of predicted and surrogate maps—yielded the best performance compared to standard metrics like MSE or MMD.
Experimental Evaluation
The method was evaluated across three domains: VQA-v1 (multi-modal), CIFAR-10 (vision), and TREC-6 (text). VQA-HAT was used specifically to measure correlation with human attention.
Key Results
- Performance Gains: The self-supervised model achieved a 3.85% accuracy increase in VQA, a 3.7% boost on TREC-6, and a 1.5% improvement on CIFAR-10.
- Human Alignment: On the VQA-HAT dataset, the model showed higher Rank Correlation (RC) and lower Earth Mover Distance (EMD), indicating its attention maps align more closely with human gaze than the baseline.
- Qualitative Improvements: Visualizations indicate that while baseline attention is often diffused, the self-supervised version produces "sharper" maps that tightly localize relevant objects, such as focusing specifically on a bowl of soup rather than the entire table.
Conclusion
This work demonstrates that a model's latent knowledge can be extracted through systematic masking to provide its own supervision. By sharpening attention maps, the method improves both performance and interpretability across modalities. Future work may address the computational cost of generating surrogate maps and explore dependencies on initial baseline quality.
Summary updated