Attention Shifting to Pursue Optimal Representation for Adapting Multi-granularity Tasks

Gairui Bai, Wei Xi, Yihan Zhao, Xinhui Liu, Jizhong Zhao

Abstract

Object recognition in open environments, e.g., video surveillance, poses significant challenges due to the inclusion of unknown and multi-granularity tasks (MGT). However, recent methods exhibit limitations as they struggle to capture subtle differences between different parts within an object and adaptively handle MGT. To address this limitation, this paper proposes a Class-semantic Guided Attention Shift (SegAS) method. SegAS transforms adaptive MGT into dynamic combinations of invariant discriminant representations across different levels to effectively enhance adaptability to multi-granularity downstream tasks. Specifically, SegAS incorporates a hardness-based Attention Part Filtering Strategy (ApFS) to dynamically decompose objects into complementary parts based on the object structure and relevance to the instance. Then, SegAS shifts attention to the optimal discriminant region of each part under the guidance of hierarchical class semantics. Finally, a diversity loss is employed to emphasize the importance and distinction of different partial features. Extensive experiments validate SegAS' effectiveness in multi-granularity recognition of three tasks.

Introduction: The Challenge of Multi-granularity Tasks

Most self-supervised learning (SSL) methods rely on instance-level consistency, attempting to align different augmented views of the same image in a feature space. While effective for general object recognition, this approach often ignores the internal semantic diversity of an object. In real-world applications like autonomous driving or video surveillance, models frequently encounter Multi-granularity Tasks (MGT). A model might need to recognize a broad category (e.g., "car") while simultaneously identifying specific fine-grained details or detecting the object when it is partially occluded.

Multi-granularity challenges in video surveillance and autonomous driving.

Standard SSL methods struggle in these scenarios because they often collapse onto a single, most-obvious feature. If that primary feature is blocked by an obstacle, the model fails. SegAS addresses this by forcing the model to "shift its attention" to secondary, more subtle discriminative regions.

Methodology: Class-semantic Guided Attention Shift (SegAS)

The SegAS framework introduces three main innovations to the standard Siamese network architecture to handle the complexity of MGT:

1. Attention Part Filtering Strategy (ApFS)

ApFS is the core mechanism for forced exploration. The model generates a heatmap of the most salient region of an object and then masks it out. This compels the model to look at the remaining parts of the object to find secondary discriminative features, building a more robust and comprehensive representation.

2. Prototype-based Consistency Regularization (ProCR)

Instead of simple image-to-image comparisons, SegAS compares image embeddings to a hierarchy of "prototypes" (cluster centers representing different semantic levels). This hierarchical guidance allows the model to learn representations that are effective across multiple levels of granularity, from broad classes to fine details.

3. Hardness Adaptive Mechanism

To ensure training stability, SegAS uses an entropy-based confidence score to determine the "hardness" of a sample. It prioritizes learning from "easy" samples in the initial phases before moving to more complex part-filtering for harder, more ambiguous instances.

4. Dual Siamese Networks and Diversity Loss

The architecture utilizes two branches to reconcile global consistency with local part differentiation. A diversity loss is added to ensure that the features learned from different parts of the object are distinct and complementary, preventing redundant feature learning.

Experimental Results

The researchers evaluated SegAS across several standard benchmarks, including ImageNet-100/1K, CUB-200-2011 (birds), and COCO/Pascal VOC for transfer learning.

Robustness and Fine-Grained Accuracy

SegAS demonstrated substantial improvements in occlusion robustness. On ImageNet-100 with random masks, SegAS achieved 83.21% accuracy, compared to MoCo v2’s 70.76%. In fine-grained classification on the CUB-200-2011 bird dataset, SegAS reached 29.70% Top-1 accuracy, a significant leap over the previous state-of-the-art HCSC (20.28%).

Localization and Transfer Learning

Qualitative results using Class Activation Maps (CAM) show that SegAS identifies multiple distinct parts of an object (e.g., the head, body, and tail of a bird) rather than just one central point, leading to a 4.26% improvement in localization accuracy. Furthermore, SegAS outperformed competing methods in downstream tasks like object detection on COCO (42.1 AP) and PASCAL VOC (82.93 AP50).

Conclusion

SegAS provides a robust framework for self-supervised learning in complex, multi-granularity environments. By intentionally shifting attention away from dominant features and utilizing hierarchical semantic guidance, the model learns a richer set of features that are resilient to occlusion and effective for fine-grained tasks. While the computational cost is slightly higher than standard baselines, the performance gains in real-world scenarios like autonomous driving and surveillance are significant. Future work may involve adapting SegAS for Vision Transformer (ViT) architectures to further exploit patch-based attention mechanisms.

Summary updated

Abstract

Object recognition in open environments, e.g., video surveillance, poses significant challenges due to the inclusion of unknown and multi-granularity tasks (MGT). However, recent methods exhibit limitations as they struggle to capture subtle differences between different parts within an object and adaptively handle MGT. To address this limitation, this paper proposes a Class-semantic Guided Attention Shift (SegAS) method. SegAS transforms adaptive MGT into dynamic combinations of invariant discriminant representations across different levels to effectively enhance adaptability to multi-granularity downstream tasks. Specifically, SegAS incorporates a hardness-based Attention Part Filtering Strategy (ApFS) to dynamically decompose objects into complementary parts based on the object structure and relevance to the instance. Then, SegAS shifts attention to the optimal discriminant region of each part under the guidance of hierarchical class semantics. Finally, a diversity loss is employed to emphasize the importance and distinction of different partial features. Extensive experiments validate SegAS' effectiveness in multi-granularity recognition of three tasks.
BibTeX
@inproceedings{bai2024attention,
  author = {Gairui Bai and Wei Xi and Yihan Zhao and Xinhui Liu and Jizhong Zhao},
  title = {{Attention Shifting to Pursue Optimal Representation for Adapting Multi-granularity Tasks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=wGHbcbHbyV},
}
Figure