CDS: Cross-Domain Self-supervised Pre-training

Donghyun Kim, Kuniaki Saito, Tae-Hyun Oh, Bryan A. Plummer, Stan Sclaroff, Kate Saenko

Abstract

CDS (Cross-Domain Self-supervised Pre-training) is a novel two-stage approach designed to enhance the generalization of deep learning models across disparate visual domains. By incorporating a targeted self-supervised pre-training step using unlabeled data from multiple target domains, CDS learns features that are simultaneously discriminative and domain-invariant. This method significantly outperforms standard ImageNet pre-training on critical tasks such as cross-domain image retrieval and universal domain adaptation.

Introduction

Deep learning models frequently encounter performance degradation when applied to data outside their training distribution, a phenomenon known as domain shift. For instance, a model trained on high-resolution photographs (the "Real" domain) often fails to accurately identify the same objects in artistic renderings or sketches.

Overview of the CDS two-stage pre-training pipeline and its application in unsupervised cross-domain image retrieval.

While standard practice involves initializing models with ImageNet pre-trained weights, these features are often biased toward the specific textures and backgrounds of the source dataset. Naively applying Self-Supervised Learning (SSL) to bridge this gap can be counterproductive; models may learn to distinguish images based on their stylistic domain rather than their semantic content, effectively widening the domain gap.

Methodology

The proposed CDS framework utilizes a two-stage pipeline. Following standard supervised pre-training on ImageNet, the model undergoes Cross-Domain Self-supervision (CDS) using unlabeled data from the downstream domains of interest (e.g., Real and Painting).

CDS optimizes two primary objectives:

  1. In-domain Instance Discrimination: The model is trained to distinguish an image from others within the same domain. By restricting comparisons to the same style, the model is forced to prioritize semantic features over domain-specific textures.
  2. Cross-domain Matching (CDM): This alignment step calculates the similarity between an image in one domain and a memory bank of features from another. By minimizing the entropy of this similarity distribution, the model is encouraged to find confident matches across domains, aligning the representations of semantically identical objects regardless of their visual style.

Key Insights and Ideas

Conceptual comparison between naive instance discrimination and the proposed CDS method.

The "Naive SSL" Trap

Standard SSL methods like SimCLR or MoCo often treat semantically similar images from different domains as "negative pairs." This pushes their representations apart in the feature space, reinforcing the domain gap. CDS avoids this by explicitly modeling cross-domain relationships.

Entropy as an Aligner

Using entropy minimization for cross-domain matching provides a robust unsupervised alignment mechanism. It requires no labels, simply tasking the model with finding a confident correlate in the alternate domain. This process shifts the model's focus from texture to shape, which is more consistent across domains.

Key Results

CDS demonstrated substantial performance gains across multiple benchmarks, including CUB-200-2011, Office-Home, and Office-31:

  • Unsupervised Image Retrieval: On the CUB dataset, CDS achieved a Precision@1 of 40.9%, nearly doubling the 22.6% achieved by standard ImageNet pre-training.
  • Universal Domain Adaptation: Integrated with the DANCE adaptation method, CDS improved the H-score (balancing accuracy on known and unknown classes) on Office-Home from 49.2 to 70.8.
  • Few-Shot Learning: In 1-shot scenarios on the CUB dataset, CDS provided a significantly stronger starting point, with accuracy jumping from baseline levels of 5-12% to over 20-28%.

Conclusion

CDS effectively bridges the domain gap by leveraging unlabeled target data to learn domain-invariant semantic features. While currently focused on domain pairs, future work may explore scaling CDS to handle dozens of domains simultaneously or integrating it with larger foundation models like CLIP or DINO to further enhance visual representation robustness.

Summary updated

Abstract

We present a two-stage pre-training approach that improves the generalization ability of standard single-domain pre-training. While standard pre-training on a single large dataset (such as ImageNet) can provide a good initial representation for transfer learning tasks, this approach may result in biased representations that impact the success of learning with new multi-domain data (e.g., different artistic styles) via methods like domain adaptation. We propose a novel pre-training approach called Cross-Domain Self-supervision (CDS), which directly employs unlabeled multi-domain data for downstream domain transfer tasks. Our approach uses self-supervision not only within a single domain but also across domains. In-domain instance discrimination is used to learn discriminative features on new data in a domain-adaptive manner, while cross-domain matching is used to learn domain-invariant features. We apply our method as a second pre-training step (after ImageNet pre-training), resulting in a significant target accuracy boost to diverse domain transfer tasks compared to standard one-stage pre-training.
BibTeX
@inproceedings{kim2021cds,
  author = {Donghyun Kim and Kuniaki Saito and Tae-Hyun Oh and Bryan A. Plummer and Stan Sclaroff and Kate Saenko},
  title = {{CDS: Cross-Domain Self-supervised Pre-training}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00899},
}
Figure