Universal Domain Adaptation through Self Supervision

Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Kate Saenko

This article summarizes the paper "Universal Domain Adaptation through Self-Supervision" by Saito et al., presented at NeurIPS 2020. The authors introduce a novel framework designed to handle arbitrary category shifts in domain adaptation without requiring prior knowledge of the shift type.

Introduction

Domain Adaptation (DA) is the art of training a model on a labeled "source" dataset (e.g., synthetic images) and making it work on an unlabeled "target" dataset (e.g., real-world photos).

Traditional DA methods are usually specialists designed for specific scenarios:

  • Closed-set: Source and target share all classes.
  • Partial: Target classes are a subset of source classes.
  • Open-set: Target contains "unknown" classes not seen in the source.

The "Universal" challenge arises because we rarely know how the categories overlap between these two domains in the real world. If you apply a closed-set method to an open-set problem, the model will hallucinate and force "unknown" objects into "known" categories.

Conceptual overview of DANCE, illustrating Neighborhood Clustering and Entropy Separation.

To address this, the authors propose DANCE (Domain Adaptive Neighborhood Clustering via Entropy optimization), a category-shift-agnostic approach that works out-of-the-box, regardless of whether the target domain has more, fewer, or different classes than the source. By combining self-supervised neighborhood clustering with an entropy-based rejection mechanism, DANCE prevents the "catastrophic misalignment" common in traditional domain adaptation methods.

The DANCE Methodology

DANCE relies on two primary novel components to achieve universal domain adaptation:

1. Neighborhood Clustering (NC)

Since the model cannot rely on source labels to understand "unknown" target categories, it uses self-supervision. DANCE maintains a memory bank of target features. For every target sample, it calculates a similarity distribution against all other samples and source prototypes. By minimizing the entropy of this distribution, the model forces each target point to move closer to its nearest neighbor (either a source class prototype or another target sample). This discovers the inherent structure of the target domain without needing labels.

2. Entropy Separation (ES) Loss

This acts as a "gatekeeper" to decide which samples belong to known source classes and which are "unknown." It exploits the fact that "unknown" samples typically result in higher entropy (uncertainty) in the source classifier. The ES loss pushes samples with low entropy even lower (aligning them with source classes) and pushes samples with high entropy even higher (rejecting them as unknown). A "confidence threshold" is used to ignore ambiguous samples near the boundary, making the learning process more robust.

Detailed mechanism of DANCE showing the memory bank, similarity distribution, and entropy-based boundary.

Additionally, the model uses Domain-Specific Batch Normalization, a "weak" alignment technique that handles style differences between domains without the risks of aggressive global distribution matching.

Key Insights and Results

The authors evaluated DANCE on several standard and large-scale benchmarks, including Office (31 classes), OfficeHome (65 classes), VisDA, and Caltech-256 & ImageNet-1K.

Key Insights: * Self-Supervision is the "Unknown" Solution: We don't need labels to handle new classes. By simply encouraging the model to cluster similar things together (NC), the model learns discriminative features for categories it has never officially "seen." * The Entropy Gap: Entropy is a powerful natural signal for domain membership. Instead of trying to force-align distributions (which often fails in universal settings), DANCE simply "amplifies" the natural uncertainty of the model to create a clear boundary between knowns and unknowns. * Agnosticism as a Strength: DANCE was the only method in the study that consistently outperformed the "Source Only" baseline across every scenario. Specialized models often performed worse than a simple non-adapted model when their category-shift assumptions were violated.

Key Results: * Rank 1 Performance: DANCE achieved an average rank of 1.2 across all scenarios (Closed, Partial, Open, Open-Partial), significantly outperforming the next best baseline, ETN (rank 2.9). * Superior Accuracy: On the OfficeHome dataset, DANCE achieved an average accuracy of 77.3%, compared to 62.0% for UAN (a previous Universal DA leader) and 61.7% for a Source-Only model. * Qualitative Success: t-SNE visualizations showed that DANCE successfully grouped "unknown" samples into distinct clusters far away from "known" source clusters, even though the model was never told how many unknown classes existed.

Limitations and Future Directions

While highly effective, DANCE does have some limitations. The model exhibits hyperparameter sensitivity; while the authors kept hyperparameters mostly consistent, the entropy threshold ($\rho$) and the confidence margin ($m$) are critical. In extremely noisy domains, finding the "perfect" entropy boundary might still require some trial and error. Furthermore, the use of a memory bank to store features for all target samples adds a computational overhead that scales with the size of the target dataset.

Looking forward, the authors suggest that this neighborhood-based approach could be extended to more complex tasks like semantic segmentation or class-incremental learning, where new categories appear over time.

Summary updated

Abstract

Unsupervised domain adaptation methods traditionally assume that all source categories are present in the target domain. In practice, little may be known about the category overlap between the two domains. While some methods address target settings with either partial or open-set categories, they assume that the particular setting is known a priori. We propose a more universally applicable domain adaptation approach that can handle arbitrary category shift, called Domain Adaptative Neighborhood Clustering via Entropy optimization (DANCE). Our approach combines two novel ideas: First, as we cannot fully rely on source categories to learn features discriminative for the target, we propose a novel neighborhood clustering technique to learn the structure of the target domain in a self-supervised way. Second, we use entropy-based feature alignment and rejection to align target features with the source, or reject them as unknown categories based on their entropy. We show through extensive experiments that DANCE outperforms baselines across open-set, open-partial and partial domain adaptation settings.
BibTeX
@inproceedings{saito2020universal,
  author = {Kuniaki Saito and Donghyun Kim and Stan Sclaroff and Kate Saenko},
  title = {{Universal Domain Adaptation through Self Supervision}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=2gVW3RLTDq},
}
Figure