MeGA-CDA: Memory Guided Attention for Category-Aware Unsupervised Domain Adaptive Object Detection

Vibashan Vs, Vikram Gupta, Poojan Oza, Vishwanath A. Sindagi, Vishal M. Patel

Abstract

Existing approaches for unsupervised domain adaptive object detection perform feature alignment via adversarial training. While these methods achieve reasonable improvements in performance, they typically perform category-agnostic domain alignment, thereby resulting in negative transfer of features. To overcome this issue, in this work, we attempt to incorporate category information into the domain adaptation process by proposing Memory Guided Attention for Category-Aware Domain Adaptation (MeGA-CDA). The proposed method consists of employing category-wise discriminators to ensure category-aware feature alignment for learning domain-invariant discriminative features. However, since the category information is not available for the target samples, we propose to generate memory-guided category-specific attention maps which are then used to route the features appropriately to the corresponding category discriminator. The proposed method is evaluated on several benchmark datasets and is shown to outperform existing approaches.

Introduction

Deep learning models for object detection are notoriously sensitive to "domain shift." A detector trained on clear, sunny city streets (the source domain) often performs poorly when deployed in foggy environments or different cities (the target domain). Unsupervised Domain Adaptation (UDA) aims to fix this without expensive manual labeling.

Standard UDA techniques typically use "global alignment," attempting to make the overall distribution of features from the target domain look like the source domain. However, global alignment is category-agnostic; it does not distinguish between a car and a tree. This often leads to negative transfer, where the model accidentally aligns the features of a car in the source domain with a building in the target domain. MeGA-CDA addresses this by moving from global alignment to category-aware alignment.

Conceptual comparison between Global Domain Adaptation and the proposed Category-Aware approach, illustrating how class-specific alignment prevents negative transfer.

Methodology

The MeGA-CDA framework is built on top of a standard Faster R-CNN architecture and introduces three novel components to enable semantic awareness during adaptation:

  1. Category-Specific Memory Modules: The model maintains $K$ memory banks (one for each object category). These banks store "prototypes"—representative feature vectors—of objects seen in the labeled source data, acting as anchors for the adaptation process.
  2. Memory-Guided Attention: Since the target domain lacks labels, the model uses the memory banks as a reference. It queries target features against these prototypes to generate attention maps. These maps highlight regions in a target image that likely contain a specific category (e.g., identifying areas that look like a "car").
  3. Category-Wise Discriminators (CDA): Instead of one global discriminator, MeGA-CDA employs $K$ separate discriminators. The attention maps act as "routers," sending features from specific regions to their corresponding category discriminator. This forces the model to perform explicit "car-to-car" and "person-to-person" alignment across domains.
The MeGA-CDA architecture, showing the integration of Memory-Guided Attention and Category-Aware Domain Adaptation modules within the detection pipeline.

To further refine this, the authors use Metric Learning. Rather than relying on simple mathematical similarity (like cosine similarity), they train a sub-network to learn the best similarity metric for generating attention maps, supervised by the ground-truth labels available in the source domain.

Experimental Results

The authors evaluated MeGA-CDA on several standard benchmarks for domain adaptive object detection, consistently outperforming existing state-of-the-art methods:

  • Foggy-Cityscapes (Adverse Weather): Achieved 41.8 mAP, a significant jump over the previous best of 39.3 mAP.
  • Sim10k to Cityscapes (Synthetic to Real): Improved performance to 44.8 mAP (vs. 43.0 mAP for the previous leader).
  • KITTI to Cityscapes (Cross-Camera): Reached 43.0 mAP, outperforming prior methods like ATF and Selective DA.

Qualitatively, the attention maps generated by the "Learned Similarity" approach were much cleaner and more focused on actual objects compared to standard cosine similarity, which often highlighted irrelevant background pixels.

Conclusion and Future Work

MeGA-CDA effectively mitigates negative transfer by leveraging category-specific memory to guide domain alignment. While the use of $K$ memory modules and discriminators increases computational overhead, the resulting improvements in detection accuracy across domains are substantial. Future work could involve exploring more efficient ways to scale this category-aware alignment to datasets with hundreds of classes or extending the memory mechanism to handle temporal consistency in video domain adaptation.

Summary updated

Abstract

Existing approaches for unsupervised domain adaptive object detection perform feature alignment via adversarial training. While these methods achieve reasonable improvements in performance, they typically perform category-agnostic domain alignment, thereby resulting in negative transfer of features. To overcome this issue, in this work, we attempt to incorporate category information into the domain adaptation process by proposing Memory Guided Attention for Category-Aware Domain Adaptation (MeGA-CDA). The proposed method consists of employing category-wise discriminators to ensure category-aware feature alignment for learning domain-invariant discriminative features. However, since the category information is not available for the target samples, we propose to generate memory-guided category-specific attention maps which are then used to route the features appropriately to the corresponding category discriminator. The proposed method is evaluated on several benchmark datasets and is shown to outperform existing approaches.
BibTeX
@inproceedings{vs2021mega,
  author = {Vibashan VS and Vikram Gupta and Poojan Oza and Vishwanath A. Sindagi and Vishal M. Patel},
  title = {{MeGA-CDA: Memory Guided Attention for Category-Aware Unsupervised Domain Adaptive Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=iuZMuxMXpP},
}
Figure