Seeking Similarities over Differences: Similarity-based Domain Alignment for Adaptive Object Detection

Farzaneh Rezaeianaran, Rakshith Shetty, Rahaf Aljundi, Daniel Olmeda Reino, Shanshan Zhang, Bernt Schiele

Abstract

In order to robustly deploy object detectors across a wide range of scenarios, they should be adaptable to shifts in the input distribution without the need to constantly annotate new data. This has motivated research in Unsupervised Domain Adaptation (UDA) algorithms for detection. UDA methods learn to adapt from labeled source domains to unlabeled target domains, by inducing alignment between detector features from source and target domains. Yet, there is no consensus on what features to align and how to do the alignment. In our work, we propose a framework that generalizes the different components commonly used by UDA methods laying the ground for an in-depth analysis of the UDA design space. Specifically, we propose a novel UDA algorithm, ViSGA, a direct implementation of our framework, that leverages the best design choices and introduces a simple but effective method to aggregate features at instance-level based on visual similarity before inducing group alignment via adversarial training. We show that both similarity-based grouping and adversarial training allows our model to focus on coarsely aligning feature groups, without being forced to match all instances across loosely aligned domains. Finally, we examine the applicability of ViSGA to the setting where labeled data are gathered from different sources. Experiments show that not only our method outperforms previous single-source approaches on Sim2Real and Adverse Weather, but also generalizes well to the multi-source setting.

Introduction

Object detectors often suffer a massive performance drop when deployed in environments different from their training data—a phenomenon known as domain shift. This occurs when moving from synthetic simulations to real-world streets or from clear weather to fog. While Unsupervised Domain Adaptation (UDA) aims to address this without requiring new manual annotations, the field lacks consensus on the granularity of alignment (e.g., aligning every proposal vs. category prototypes) and the underlying mechanism (e.g., adversarial training vs. contrastive learning). Current methods often rely on noisy pseudo-labels, which can lead to error propagation.

The ViSGA Framework

The authors propose ViSGA (Visually Similar Group Alignment), which focuses on "similarity over differences." Its architecture is built on top of a Faster R-CNN detector and introduces a novel grouping strategy to facilitate domain-invariant feature learning.

Similarity-based Grouping

Instead of using predicted class labels or spatial overlap (IoU), ViSGA uses hierarchical agglomerative clustering. It groups object proposals based on the cosine distance of their feature embeddings. This allows the model to group similar-looking objects even if they are far apart in the image, providing a "middle ground" that captures intra-domain variance without being overwhelmed by outliers.

Conceptual overview of similarity-based grouping for domain alignment.

Architecture and Alignment

The grouping process is dynamic and class-agnostic, adapting to scene complexity without relying on potentially incorrect category predictions. Once groups are formed, ViSGA employs a Gradient Reversal Layer (GRL) and a discriminator to perform adversarial training at both the image level and the group level. This forces the network to learn features that are useful for detection but "domain-blind."

The ViSGA architecture integrated with Faster R-CNN, showing image-level and instance-level alignment.

Experimental Setup

ViSGA was evaluated across several challenging domain shift scenarios:

  • Sim2Real: Training on SIM10k (synthetic) and adapting to Cityscapes (real-world).
  • Adverse Weather: Training on Cityscapes (clear) and adapting to Foggy Cityscapes.
  • Cross Camera: Training on KITTI and adapting to Cityscapes.
  • Multi-Source: Adapting to Cityscapes using data from multiple sources (KITTI + SIM10k).

Key Results

ViSGA achieved state-of-the-art (SOTA) results across all benchmarks:

  • Sim2Real: Achieved 49.3% mAP, outperforming the previous best (GPA) at 47.6%.
  • Foggy Weather: Achieved 43.3% mAP, a significant jump over the 39.5% achieved by prior methods.
  • Multi-Source: Reached 51.3% mAP, demonstrating effective scaling with diverse labeled data.

Ablation studies confirmed that removing the similarity-based grouping caused performance to drop by several percentage points, highlighting it as the core driver of the model's success.

Discussion and Conclusion

The paper finds that Adversarial Training (AT) consistently outperforms Contrastive Learning (CL) in this context, as AT allows the network more freedom to align feature distributions naturally. By ignoring class labels during the grouping process, ViSGA avoids the "vicious cycle" where initial wrong guesses on the target domain ruin the alignment process.

While effective, the hierarchical clustering step adds a small computational overhead during training. Additionally, the "cluster radius" parameter requires tuning based on the severity of the domain gap. Future directions include using similarity-based groups to improve pseudo-label quality or applying the framework to datasets where category definitions shift between domains.

Summary updated

Abstract

In order to robustly deploy object detectors across a wide range of scenarios, they should be adaptable to shifts in the input distribution without the need to constantly an-notate new data. This has motivated research in Unsupervised Domain Adaptation (UDA) algorithms for detection. UDA methods learn to adapt from labeled source domains to unlabeled target domains, by inducing alignment between detector features from source and target domains. Yet, there is no consensus on what features to align and how to do the alignment. In our work, we propose a framework that generalizes the different components commonly used by UDA methods laying the ground for an in-depth analysis of the UDA design space. Specifically, we propose a novel UDA algorithm, ViSGA, a direct implementation of our framework, that leverages the best design choices and introduces a simple but effective method to aggregate features at instance-level based on visual similarity before inducing group alignment via adversarial training. We show that both similarity-based grouping and adversarial training allows our model to focus on coarsely aligning feature groups, without being forced to match all instances across loosely aligned domains. Finally, we examine the applicability of ViSGA to the setting where labeled data are gathered from different sources. Experiments show that not only our method outperforms previous single-source approaches on Sim2Real and Adverse Weather, but also generalizes well to the multi-source setting.
BibTeX
@inproceedings{rezaeianaran2021seeking,
  author = {Farzaneh Rezaeianaran and Rakshith Shetty and Rahaf Aljundi and Daniel Olmeda Reino and Shanshan Zhang and Bernt Schiele},
  title = {{Seeking Similarities over Differences: Similarity-based Domain Alignment for Adaptive Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=TJXhMydKWu2},
}
Figure