Improving Object Detection with Inverted Attention

Zeyi Huang, Wei Ke, Dong Huang

Abstract

Improving object detectors against occlusion, blur and noise is a critical step to deploy detectors in real applications. Since it is not possible to exhaust all image defects and occlusions through data collection, many researchers seek to generate occluded samples. The generated hard samples are either images or feature maps with coarse patches dropped out in the spatial dimensions. Significant overheads are required in generating hard samples and/or estimating drop-out patches using extra network branches. In this paper, we improve object detectors using a highly efficient and fine-grain mechanism called Inverted Attention (IA). Different from the original detector network that only focuses on the dominant part of objects, the detector network with IA iteratively inverts attention on feature maps which pushes the detector to discover new discriminative clues and puts more attention on complementary object parts, feature channels and even context. Our approach (1) operates along both the spatial and channels dimensions of the feature maps; (2) requires no extra training on hard samples, no extra network parameters for attention estimation, and no testing overheads. Experiments show that our approach consistently improved state-of-the-art detectors on benchmark databases.

Introduction

Deep learning-based object detectors often exhibit a "lazy" behavior: they rely on a small subset of highly discriminative features to identify objects. For instance, a model might learn to recognize a car primarily by its wheels. While effective in clear conditions, this reliance makes models fragile when faced with real-world challenges like occlusion, blur, or noise. If the primary feature is obscured, the detector fails.

Standard detectors often focus on a single discriminative part, such as a wheel, making them vulnerable to occlusion.

To address this, researchers have explored generating "hard samples" by artificially occluding images. However, these methods typically involve complex adversarial training or additional network branches, leading to significant computational overhead. This paper proposes Inverted Attention (IA), a fine-grained, efficient training-time mechanism that forces models to explore the entire object and its context without any inference-time cost.

The Inverted Attention (IA) Mechanism

The IA module is a "plug-in" active only during the training phase. It leverages the model's own gradients to identify and temporarily suppress the most influential features, forcing the network to find alternative discriminative clues.

The process follows four key steps:

  1. Gradient-Guided Attention: The model calculates gradients of feature maps relative to the classification score. High gradients highlight the regions the model currently relies on most.
  2. Inversion: The module identifies these "sensitive" spatial areas and feature channels and suppresses them (e.g., by zeroing them out).
  3. Feature Refinement: The original feature maps are masked by this inverted attention. This forces the back-propagation to update weights based on previously ignored parts of the object.
  4. Dual-Dimension Operation: IA operates across both spatial and channel dimensions, ensuring a diverse set of internal filters are developed.
Visualization of inverted attention masks during training (top) and the resulting diversified attention maps (middle) compared to baseline.

Because IA is removed during testing, the final model benefits from enhanced feature diversity with zero additional computational cost during deployment.

Experimental Results

The IA module was evaluated on standard benchmarks including PASCAL VOC 2007/2012 and MS-COCO using popular architectures like Fast R-CNN, Faster R-CNN, and Mask R-CNN.

Performance Gains

  • PASCAL VOC 2012: Faster R-CNN (ResNet101) saw a substantial improvement, jumping from 73.8% to 79.2% mAP.
  • MS-COCO: IA provided a consistent boost of 1.1 to 2.1 AP points across various ResNet backbones.
  • Localization Precision: Performance gains were particularly notable at higher Intersection over Union (IoU) thresholds, indicating that the predicted bounding boxes were more accurately aligned with the objects.

Key Insights

  • The "Backup Plan": By hiding primary features during training, IA forces the model to develop "backup" features (e.g., recognizing a bird by its wings if the beak is hidden).
  • Contextual Awareness: IA often pushes the model to utilize surrounding context, further improving robustness in complex scenes.
  • Efficiency: Unlike adversarial methods, IA uses existing gradients, requiring no extra parameters or training stages.

Conclusion and Future Work

Inverted Attention offers a simple yet powerful way to improve the robustness and accuracy of object detectors. By discouraging "lazy" feature reliance, it produces models that generalize better to occlusions and noise. While the authors noted that applying IA to all training samples can lead to overfitting—suggesting a 20% application rate as an optimal balance—the method remains highly effective. Future research may extend the IA principle to other domains such as instance segmentation and object tracking.

Summary updated

Abstract

Improving object detectors against occlusion, blur and noise is a critical step to deploy detectors in real applications. Since it is not possible to exhaust all image defects and occlusions through data collection, many researchers seek to generate occluded samples. The generated hard samples are either images or feature maps with coarse patches dropped out in the spatial dimensions. Significant overheads are required in generating hard samples and/or estimating drop-out patches using extra network branches. In this paper, we improve object detectors using a highly efficient and fine-grain mechanism called Inverted Attention (IA). Different from the original detector network that only focuses on the dominant part of objects, the detector network with IA iteratively inverts attention on feature maps which pushes the detector to discover new discriminative clues and puts more attention on complementary object parts, feature channels and even context. Our approach (1) operates along both the spatial and channels dimensions of the feature maps; (2) requires no extra training on hard samples, no extra network parameters for attention estimation, and no testing overheads. Experiments show that our approach consistently improved state-of-the-art detectors on benchmark databases.
BibTeX
@inproceedings{huang2020improving,
  author = {Zeyi Huang and Wei Ke and Dong Huang},
  title = {{Improving Object Detection with Inverted Attention}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=6O2CQrmZwZY},
}
Figure