Towards Online Domain Adaptive Object Detection

Vibashan Vs, Poojan Oza, Vishal M. Patel

TL;DR

This paper introduces a novel framework for object detection that can adapt to new visual domains during deployment (online) without access to the original training data. By leveraging a memory-enhanced contrastive learning approach, their method significantly improves robustness to domain shifts, outperforming existing offline and online adaptation techniques.

What Problem Does This Paper Address?

Deep learning models, especially for object detection, often perform poorly when deployed in real-world scenarios that differ from their training data. This "domain shift" can be caused by variations in lighting, weather, camera angles, or even entirely new environments. Traditional Unsupervised Domain Adaptation (UDA) methods require both labeled source data and unlabeled target data during training, which is often impractical or impossible in real-world deployment due to privacy or data availability constraints. More recent Source-Free Domain Adaptation (SFDA) methods address this by adapting a pre-trained model to a target domain without source data, but they typically operate offline. This means the model is adapted once before deployment. However, real-world applications can encounter a continuous stream of new domain shifts, making offline adaptation insufficient. This paper tackles the critical challenge of online domain adaptation for object detection, where the model must continuously adapt to new domains in real-time as it encounters new data, without access to the original source data.

Methods

The core of the proposed solution is a unified framework that supports both offline and online adaptation. Key components include:

  • Student-Teacher Framework: To prevent catastrophic forgetting of learned knowledge during online adaptation, a student-teacher architecture is employed. The student model learns from pseudo-labels generated by a teacher model, and the teacher model is updated via Exponential Moving Average (EMA) of the student's weights. This ensures more stable pseudo-labels.
  • MemXformer Module: This is the novel contribution. It's a cross-attention transformer-based memory module designed to store "prototypical patterns" of the target domain distribution.
  • Write Operation: Teacher network's Region of Interest (RoI) features are used to update the memory bank. Cross-attention between teacher RoI features and memory items determines how memory elements are updated, effectively capturing target domain characteristics.
  • Read Operation: Student network's RoI features query the memory bank. A weighted sum of similar memory elements is retrieved, providing strong positive pairs for contrastive learning. This operation also uses cross-attention.
  • Memory-based Contrastive Learning (MemCLR): This loss function leverages the MemXformer. It uses the retrieved strong positive pairs (from the memory bank) and mined negative pairs to guide the student model in learning robust target-specific feature representations. This is more efficient than traditional contrastive learning methods that require large batch sizes and heavy augmentations, especially for object detection.
  • Negative Pair Mining: To further refine contrastive learning, the method identifies and uses the least similar memory items as negative pairs, preventing the model from being misled by slightly dissimilar but highly attended memory elements.

The overall loss combines the pseudo-label supervision loss and the MemCLR loss.

Datasets

The paper evaluates its method on several standard domain adaptation benchmarks:

  • Cityscapes → FoggyCityscapes: Adapting from clear weather to foggy conditions.
  • Sim10K → Cityscapes: Adapting from synthetic car images to real-world urban scenes.
  • KITTI → Cityscapes: Adapting across different camera setups.
  • PASCAL-VOC → Watercolor: Adapting from real-world objects to artistic renderings (concept shift).

Key Insights and Ideas

  • Online Adaptation is Crucial for Real-World Deployment: The paper convincingly argues that static, offline adaptation is insufficient for dynamic real-world environments where domain shifts are continuous and unpredictable.
  • Memory as a Dynamic Target Domain Representation: The MemXformer's ability to store and retrieve prototypical patterns of the target domain is key. It acts as a dynamic knowledge base that evolves with the incoming data.
  • Contrastive Learning Adapted for Object Detection: By using RoI features and the memory bank, the authors efficiently adapt contrastive learning principles to the object detection task, overcoming the limitations of batch size and augmentation requirements of prior methods.
  • Unified Framework for Offline and Online: The ability to perform well in both settings is a significant advantage, allowing for pre-deployment adaptation (offline) and continuous adaptation (online).

Key Results

  • State-of-the-Art Performance: The proposed method achieves state-of-the-art results on multiple domain shift scenarios for both offline and online adaptation settings, outperforming existing UDA, SFDA, and Test-Time Adaptation methods.
  • Significant Gains in Online SFDA: For instance, on the Cityscapes → FoggyCityscapes dataset, the online method outperforms SFDA methods like SFOD by 1.7 mAP and Tent by 3.0 mAP.
  • Robustness to Concept Shift: The method shows effectiveness even in challenging "real to artistic" adaptation scenarios.
  • Qualitative Improvements: t-SNE visualizations demonstrate that the proposed method learns more compact and discriminative feature representations with better classification boundaries compared to baselines.
  • Efficiency: The method takes approximately 1 second to perform online adaptation for a single sample.

Limitations and Future Work

  • Memory Bank Capacity: The paper analyzes the impact of memory bank size, finding that 1024 memory items perform best. Further exploration into optimal memory management could be beneficial.
  • Computational Cost: While efficient for online adaptation, the MemXformer adds some computational overhead. The paper notes it takes around 1 second per sample for online adaptation.
  • Future Work: The authors suggest that their unified framework can be extended to other vision tasks beyond object detection. They also highlight the potential for further improvements by exploring more sophisticated memory update and retrieval mechanisms.

Summary updated

Abstract

Existing object detection models assume both the training and test data are sampled from the same source domain. This assumption does not hold true when these detectors are deployed in real-world applications, where they encounter new visual domain. Unsupervised Domain Adaptation (UDA) methods are generally employed to mitigate the adverse effects caused by domain shift. Existing UDA methods operate in an offline manner where the model is first adapted towards the target domain and then deployed in real-world applications. However, this offline adaptation strategy is not suitable for real-world applications as the model frequently encounters new domain shifts. Hence, it becomes critical to develop a feasible UDA method that generalizes to these domain shifts encountered during deployment time in a continuous online manner. To this end, we propose a novel unified adaptation framework that adapts and improves generalization on the target domain in online settings. In particular, we introduce MemXformer - a cross-attention transformer-based memory module where items in the memory take advantage of domain shifts and record prototypical patterns of the target distribution. Further, MemXformer produces strong positive and negative pairs to guide a novel contrastive loss, which enhances target specific representation learning. Experiments on diverse detection benchmarks show that the proposed strategy can produce state-of-the-art performance in both online and offline settings. To the best of our knowledge, this is the first work to address online and offline adaptation settings for object detection. Code at https://github.com/Vibashan/memXformer-online-da
BibTeX
@inproceedings{vs2023towards,
  author = {Vibashan VS and Poojan Oza and Vishal M. Patel},
  title = {{Towards Online Domain Adaptive Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure