Self Supervision for Attention Networks

Badri N. Patro, G. S. Kasturi, Ansh Jain, Vinay P. Namboodiri

Abstract

Attention mechanisms are pivotal in modern deep learning, yet they often lack explicit supervision, leading to reliance on irrelevant background features. This paper presents a novel self-supervised approach to refine attention maps without human-annotated labels. By generating "surrogate" ground-truth attention maps through input masking and confidence probing, the proposed method distills discriminative patterns back into the network. Experimental results across vision, text, and multi-modal tasks demonstrate significant improvements in both predictive accuracy and alignment with human attention.

Introduction

While attention modules allow models to focus on salient input regions, they are typically trained implicitly. In tasks like Visual Question Answering (VQA), a model might correctly answer a question while attending to irrelevant background noise rather than the target object. This lack of grounding reduces interpretability and robustness. Although human-annotated datasets like VQA-HAT provide explicit supervision, they are prohibitively expensive to scale. The authors address this by asking: can we provide supervision to an attention module without manual labeling?

Methodology

The proposed framework introduces a two-stage pipeline for self-supervised attention.

1. Generating Surrogate Attention Maps

To identify truly discriminative regions, the authors probe a pre-trained baseline model using randomly masked versions of the input. For images, 500 random masks are applied; for text, two words are masked at a time. The model's confidence in the ground-truth class is recorded for each variation. The mask that preserves the highest confidence is designated as the "surrogate ground truth" (Att-hat). This process assumes that the regions visible in the highest-confidence mask are the most critical for the task.

Detailed workflow for generating surrogate attention maps using random masks and a selector module.

2. Training with Attention Supervision

The network is then re-trained using a multi-task loss function combining standard classification loss with an attention loss. The attention loss penalizes the discrepancy between the model’s internal attention and the surrogate map. The authors found that an adversarial loss—utilizing a discriminator to align the distributions of predicted and surrogate maps—yielded the best performance compared to standard metrics like MSE or MMD.

Overview of the Self-Supervised Attention framework for Visual Question Answering (VQA).

Experimental Evaluation

The method was evaluated across three domains: VQA-v1 (multi-modal), CIFAR-10 (vision), and TREC-6 (text). VQA-HAT was used specifically to measure correlation with human attention.

Key Results

  • Performance Gains: The self-supervised model achieved a 3.85% accuracy increase in VQA, a 3.7% boost on TREC-6, and a 1.5% improvement on CIFAR-10.
  • Human Alignment: On the VQA-HAT dataset, the model showed higher Rank Correlation (RC) and lower Earth Mover Distance (EMD), indicating its attention maps align more closely with human gaze than the baseline.
  • Qualitative Improvements: Visualizations indicate that while baseline attention is often diffused, the self-supervised version produces "sharper" maps that tightly localize relevant objects, such as focusing specifically on a bowl of soup rather than the entire table.

Conclusion

This work demonstrates that a model's latent knowledge can be extracted through systematic masking to provide its own supervision. By sharpening attention maps, the method improves both performance and interpretability across modalities. Future work may address the computational cost of generating surrogate maps and explore dependencies on initial baseline quality.

Summary updated

Abstract

In recent years, the attention mechanism has become a fairly popular concept and has proven to be successful in many machine learning applications. However, deep learning models do not employ supervision for these attention mechanisms which can improve the model's performance significantly. Therefore, in this paper, we tackle this limitation and propose a novel method to improve the attention mechanism by inducing "self-supervision". We devise a technique to generate desirable attention maps for any model that utilizes an attention module. This is achieved by examining the model's output for different regions sampled from the input and obtaining the attention probability distributions that enhance the proficiency of the model. The attention distributions thus obtained are used for supervision. We rely on the fact, that attenuation of the unimportant parts, allows a model to attend to more salient regions, thus strengthening the prediction accuracy. The quantitative and qualitative results published in this paper show that this method successfully improves the attention mechanism as well as the model's accuracy. In addition to the task of Visual Question Answering(VQA), we also show results on the task of Image classification and Text classification to prove that our method can be generalized to any vision and language model that uses an attention module.
BibTeX
@inproceedings{patro2021self,
  author = {Badri N. Patro and G. S. Kasturi and Ansh Jain and Vinay P. Namboodiri},
  title = {{Self Supervision for Attention Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00077},
}

Loading related content...

Figure