Attention Shifting to Pursue Optimal Representation for Adapting Multi-granularity Tasks

Gairui Bai, Wei Xi, Yihan Zhao, Xinhui Liu, Jizhong Zhao, Shize Guo

Abstract

Self-supervised learning (SSL) has traditionally focused on instance-level consistency, often overlooking the nuances required for multi-granularity tasks (MGT) such as fine-grained classification and recognition under occlusion. This paper presents SegAS (Class-semantic Guided Attention Shift), a novel framework designed to enhance feature representation by dynamically shifting attention away from dominant object parts. By masking the most discriminative regions during training, SegAS compels the model to discover subtle, secondary features essential for complex real-world scenarios. Evaluated on benchmarks like ImageNet and CUB-200-2011, SegAS demonstrates significant improvements in robustness and localization accuracy, outperforming existing state-of-the-art SSL methods.

Introduction: The Challenge of Multi-granularity Tasks

Standard self-supervised learning methods, such as MoCo or SimCLR, typically learn to recognize objects by focusing on their most prominent features. While effective for general classification, this approach falls short in Multi-granularity Tasks (MGT). In practical applications like autonomous driving or video surveillance, models must identify specific individuals or recognize objects despite significant occlusions.

Real-world scenarios for multi-granularity tasks in video surveillance and autonomous driving.

As shown in the figure above, real-world environments present challenges where objects are partially hidden or require fine-grained differentiation. Standard SSL tends to ignore the subtle part-level differences crucial for these tasks, focusing instead on the most obvious visual cues.

Methodology: The SegAS Framework

To address these limitations, the authors propose SegAS, which integrates three core components into a Siamese network architecture:

  1. Attention Part Filtering Strategy (ApFS): This core mechanism generates a heatmap of the most discriminative object regions and filters them out. This "attention shift" forces the twin network to identify the object using remaining, less obvious features.
  2. Hardness Adaptive Mechanism: To prevent the model from failing on extremely difficult samples, this mechanism uses an entropy-based confidence score. It modulates the intensity of filtering; as the model's confidence grows, it shifts attention more aggressively.
  3. Prototype-based Consistency Regularization (ProCR): Instead of simple instance-to-instance comparison, SegAS aligns features with "prototypes"—representative cluster centers. This facilitates the learning of hierarchical semantics, such as recognizing a specific breed within a broader category.
  4. Diversity Loss: This loss function ensures that features from original and filtered images remain distinct yet semantically relevant, preventing redundancy in the learned representations.

Experimental Results

SegAS was evaluated across multiple benchmarks, demonstrating superior generalization and robustness:

  • Occlusion Robustness: On ImageNet-100 with random masking, SegAS achieved 83.21% accuracy, a substantial lead over MoCo v2's 70.76%.
  • Fine-Grained Classification: On the CUB-200 bird species dataset, SegAS reached 29.70% Top-1 accuracy, nearly 10% higher than the previous leader, HCSC.
  • Object Localization: The framework significantly improved localization, achieving 57.3% Top-1 accuracy on ImageNet-100, compared to 47.74% for the baseline.

Conclusion and Future Directions

SegAS effectively addresses the limitations of standard SSL in multi-granularity tasks by forcing models to learn more holistic and subtle representations. While the method introduces a slight increase in computational overhead—approximately 0.5 days of additional training time on a standard 4x GPU setup—the performance gains are significant. Future research will explore adapting this attention-shifting logic to Vision Transformer (ViT) architectures to further leverage spatial attention mechanisms.

Summary updated

BibTeX
@inproceedings{bai2024attention,
  author = {Gairui Bai and Wei Xi and Yihan Zhao and Xinhui Liu and Jizhong Zhao and Shize Guo},
  title = {{Attention Shifting to Pursue Optimal Representation for Adapting Multi-granularity Tasks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=wGHbcbHbyV},
}
Figure