Towards Online Domain Adaptive Object Detection

Vibashan Vs, Poojan Oza, Vishal M. Patel

TL;DR

This paper introduces MemXformer, a novel framework for online domain adaptive object detection. It tackles the challenge of adapting object detectors to new, unseen environments as they are deployed, rather than requiring a separate offline training phase. MemXformer uses a memory module and contrastive learning to continuously improve the detector's performance in real-world, shifting conditions.

What Problem Does This Paper Address?

Modern object detection models are typically trained on large, curated datasets. However, when these models are deployed in real-world scenarios (e.g., self-driving cars, surveillance), they often encounter data that differs significantly from their training data. This "domain shift" can drastically degrade performance.

Traditional Unsupervised Domain Adaptation (UDA) methods try to bridge this gap by using both labeled source data and unlabeled target data during an offline adaptation phase. More recently, Source-Free Domain Adaptation (SFDA) emerged, allowing adaptation using only the pre-trained source model and unlabeled target data, still in an offline manner.

The critical limitation of these approaches is their offline nature. Real-world deployments face continuous, unpredictable domain shifts (e.g., changing weather, lighting, camera angles). Adapting the model only once before deployment is insufficient. This paper addresses the need for online adaptation, where the model continuously learns and improves from incoming target data during its operational lifetime, without needing access to the original source data.

Methods

The core contribution is a unified framework for both offline and online domain adaptive object detection, with a particular focus on the online setting.

  • Student-Teacher Framework: To prevent catastrophic forgetting of previously learned knowledge during online adaptation, the authors employ a student-teacher architecture. The "student" model is updated using pseudo-labels generated by the "teacher" model. The teacher's weights are then updated via an Exponential Moving Average (EMA) of the student's weights, providing a stable target for learning.
  • MemXformer Module: This is the novel component. It's a cross-attention transformer-based memory module designed to capture and store "prototypical patterns" of the target domain as it shifts over time.
  • Write Operation: Teacher network's Region of Interest (RoI) features are used to update the memory. Cross-attention between teacher RoI features and memory items helps the memory learn relevant target patterns.
  • Read Operation: Student network's RoI features are used as queries to the memory. Cross-attention retrieves weighted sums of similar memory items, effectively generating "strong positive pairs" for contrastive learning.
  • Memory-Based Contrastive Learning (MemCLR): Instead of relying on computationally expensive input augmentations (common in classification CL), this method leverages the RPN's generated RoI proposals as different "views" of an object.
  • Positive Pairs: Generated by the MemXformer's read operation, these are ensembles of similar memory items, representing temporal target distribution patterns.
  • Negative Pairs: The authors introduce "negative pair mining" by identifying the least similar memory items from the cross-attention scores, preventing the model from learning from dissimilar prototypes.
  • Loss Function: A contrastive loss is applied using these positive and negative pairs to enhance target-specific feature representation learning.

The overall loss combines the pseudo-label supervision loss and the MemCLR loss.

Datasets

The paper evaluates its method on several standard domain adaptation benchmarks:

  • Cityscapes → FoggyCityscapes: Simulates clear-weather to foggy-weather adaptation.
  • Sim10k → Cityscapes: Synthetic-to-real world adaptation.
  • KITTI → Cityscapes: Cross-camera adaptation.
  • PASCAL-VOC → Watercolor: Real-to-artistic image adaptation (concept shift).

Key Insights and Ideas

  • Online Adaptation is Crucial for Real-World Deployment: The paper strongly argues that offline adaptation is insufficient for dynamic environments. Continuous learning during deployment is necessary.
  • Memory as a Dynamic Target Representation: MemXformer's ability to store and update prototypical patterns of the target domain is key to adapting to evolving shifts. It acts as a dynamic reference.
  • Leveraging RPN Proposals for Contrastive Learning: The insight that RPN proposals can serve as diverse views for object-level contrastive learning in detectors is efficient and avoids heavy image augmentations.
  • Negative Pair Mining Enhances Robustness: Simply using similar memory items as positive pairs might still include some dissimilar information. Explicitly mining negative pairs from the memory improves the quality of contrastive signals.
  • Unified Framework for Online and Offline: The proposed method is shown to be effective in both settings, offering flexibility.

Key Results

  • State-of-the-Art Performance: The proposed MemXformer method achieves state-of-the-art results on various domain adaptation benchmarks for object detection, in both online and offline settings.
  • Significant Gains in Online SFDA: In online SFDA scenarios, the method outperforms existing UDA and SFDA methods, sometimes by substantial margins (e.g., 3-4 mAP over UDA methods, 1.7 mAP over SFDA methods on Cityscapes → FoggyCityscapes). It also significantly surpasses test-time adaptation methods like Tent.
  • Qualitative Improvements: t-SNE visualizations show that the proposed method learns more compact and discriminative RoI features, leading to better classification boundaries compared to baselines.
  • Ablation Studies: The memory bank size (1024 items) was found to be optimal. Supervised contrastive learning provided a boost, but the proposed memory-based contrastive learning was superior.

Limitations and Future Work

  • Online Adaptation Speed: While the paper notes the model takes around 1 second per sample for online adaptation, it identifies "increasing the online adaptation speed further" as a potential direction for future research. This is critical for real-time applications.
  • Memory Bank Capacity: While 1024 memory items performed best, exploring optimal capacity for different scenarios could be beneficial.
  • Generalization to Extreme Shifts: The paper demonstrates effectiveness across various shifts, but extreme, rapid, or completely novel domain shifts might still pose challenges.

Summary updated

Abstract

Existing object detection models assume both the training and test data are sampled from the same source domain. This assumption does not hold true when these detectors are deployed in real-world applications, where they encounter new visual domain. Unsupervised Domain Adaptation (UDA) methods are generally employed to mitigate the adverse effects caused by domain shift. Existing UDA methods operate in an offline manner where the model is first adapted towards the target domain and then deployed in real-world applications. However, this offline adaptation strategy is not suitable for real-world applications as the model frequently encounters new domain shifts. Hence, it becomes critical to develop a feasible UDA method that generalizes to these domain shifts encountered during deployment time in a continuous online manner. To this end, we propose a novel unified adaptation framework that adapts and improves generalization on the target domain in online settings. In particular, we introduce MemXformer - a cross-attention transformer-based memory module where items in the memory take advantage of domain shifts and record prototypical patterns of the target distribution. Further, MemXformer produces strong positive and negative pairs to guide a novel contrastive loss, which enhances target specific representation learning. Experiments on diverse detection benchmarks show that the proposed strategy can produce state-of-the-art performance in both online and offline settings. To the best of our knowledge, this is the first work to address online and offline adaptation settings for object detection. Code at https://github.com/Vibashan/memXformer-online-da
BibTeX
@inproceedings{vs2023towards,
  author = {Vibashan VS and Poojan Oza and Vishal M. Patel},
  title = {{Towards Online Domain Adaptive Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure