What Has Been Overlooked in Contrastive Source-Free Domain Adaptation: Leveraging Source-Informed Latent Augmentation within Neighborhood Context

Jing Wang, Wonho Bae, Jiahong Chen, Kuangen Zhang, Leonid Sigal, Clarence W. de Silva

TL;DR

The paper introduces SiLAN, a novel latent augmentation method that significantly improves Source-Free Domain Adaptation (SFDA). By leveraging the "memory" of a source pre-trained model to guide noise-based augmentation in the latent space, SiLAN helps models better cluster unlabeled target data, achieving state-of-the-art performance with a simple, efficient contrastive loss.

What Problem Does This Paper Address?

In many real-world scenarios, we want to adapt a high-performing AI model to a new environment (the target domain) without having access to the original training data (source domain). This is called Source-Free Domain Adaptation (SFDA). It is essential for protecting data privacy and intellectual property.

The core challenge is the "domain shift": a model trained on synthetic images (source) often struggles with real-world photos (target). While contrastive learning—a technique that teaches models to group similar things together and push different things apart—is popular for this task, existing methods often fail to generate high-quality "positive keys" (examples that should be grouped together). They either rely on simple data transformations or noisy neighborhood searches, which can lead to misaligned clusters and poor accuracy.

Figure 1: t-SNE visualization of target features extracted by the source pre-trained encoder, reveal- ing significant feature dispersion due to domain shift. However, nearby target samples still tend to share similar ground truths, motivating our in-depth exploration of contrastive clustering in SFDA.

Methods

The authors propose SiLAN (Source-informed Latent Augmented Neighborhood). Instead of just looking for nearby samples in the target domain, SiLAN uses a two-step process to create better training signals:

  1. Neighborhood Centroid: It identifies the $K$-nearest neighbors of a target sample and calculates their average position (centroid) in the latent feature space.
  2. Source-Informed Augmentation: It adds Gaussian noise to this centroid. Crucially, the variance of this noise is determined by how the original source model views those same neighbors.

By using the source model's "perspective" to define the range of the noise, the method creates a "Gaussian profile" around the sample. When trained with a standard contrastive loss (InfoNCE), this profile acts like a force field that pulls similar samples toward the correct cluster and pushes different ones away. Unlike standard data augmentation (like flipping or cropping images), this happens directly in the model's internal mathematical representation (the latent space), making it more robust and modality-independent.

Datasets

The researchers evaluated SiLAN on three widely recognized computer vision benchmarks:

  • Office-31: A standard dataset with 31 object classes across three domains (Amazon, Webcam, DSLR).
  • Office-Home: A more challenging dataset with 65 classes across four domains (Artistic, Clipart, Product, Real-World).
  • VisDA-2017: A large-scale dataset focused on the synthetic-to-real domain shift.
  • Moon Dataset: A toy "half-moon" dataset used for visualizing decision boundaries.

Key Insights and Ideas

  • Neighborhood Informativeness: The authors observed that even when a model is confused by domain shift, a sample's immediate neighbors in the feature space still tend to share the same ground-truth label. This "local consistency" is the key to successful adaptation.
  • The Pull-and-Push Effect: Latent augmentation doesn't just add variety; it mathematically stretches the boundaries of non-overlapping regions between classes. This helps the model "clean up" the messy, overlapping clusters caused by domain shift.
  • Source Knowledge is Persistent: Even without the raw source data, the structure of the source model's feature space contains valuable information about how much "spread" or variance a class should have. SiLAN captures this through the noise variance $\sigma^2_s$.
The "pull-and-push" effect of SiLAN, where latent augmentation helps reduce feature overlaps between different class clusters.

Key Results

SiLAN consistently outperformed existing state-of-the-art (SOTA) methods across all benchmarks:

  • Accuracy: On VisDA-2017, SiLAN achieved 88.3% average accuracy, surpassing complex methods like DaC and AaD.
  • Efficiency: SiLAN is significantly faster to converge. On VisDA-2017, it reached peak performance in roughly 3,727 seconds, compared to over 12,656 seconds for the previous SOTA method (DaC).
  • Versatility: The authors showed that SiLAN can be used as a "plug-and-play" augmentation module to boost the performance of other existing SFDA frameworks (like NRC or HCL) by up to 2.6%.

Limitations and Future Work

  • Hyperparameter Tuning: The method relies on choosing the right number of neighbors ($K_t$ and $K_s$). While the paper provides general guidance (e.g., smaller $K$ for smaller datasets), finding the absolute optimum still requires some empirical search.
  • Backbone Focus: While the paper demonstrates success on ResNet and Vision Transformer (ViT) architectures, the theoretical analysis assumes a linear classifier head. Exploring how this behaves with more complex, non-linear heads could be a future direction.
  • Beyond Vision: The current work focuses on image classification. The authors suggest that because latent augmentation is modality-agnostic, it could potentially be applied to NLP or audio tasks, though this hasn't been tested yet.

Summary updated

Abstract

Source-free domain adaptation (SFDA) involves adapting a model originally trained using a labeled dataset ({\em source domain}) to perform effectively on an unlabeled dataset ({\em target domain}) without relying on any source data during adaptation. This adaptation is especially crucial when significant disparities in data distributions exist between the two domains and when there are privacy concerns regarding the source model's training data. The absence of access to source data during adaptation makes it challenging to analytically estimate the domain gap. To tackle this issue, various techniques have been proposed, such as unsupervised clustering, contrastive learning, and continual learning. In this paper, we first conduct an extensive theoretical analysis of SFDA based on contrastive learning, primarily because it has demonstrated superior performance compared to other techniques. Motivated by the obtained insights, we then introduce a straightforward yet highly effective latent augmentation method tailored for contrastive SFDA. This augmentation method leverages the dispersion of latent features within the neighborhood of the query sample, guided by the source pre-trained model, to enhance the informativeness of positive keys. Our approach, based on a single InfoNCE-based contrastive loss, outperforms state-of-the-art SFDA methods on widely recognized benchmark datasets.
BibTeX
@article{wang2024what,
  author = {Jing Wang and Wonho Bae and Jiahong Chen and Kuangen Zhang and Leonid Sigal and Clarence W. de Silva},
  title = {{What Has Been Overlooked in Contrastive Source-Free Domain Adaptation: Leveraging Source-Informed Latent Augmentation within Neighborhood Context}},
  journal = {Accepted by TMLR},
  year = {2024},
  url = {https://openreview.net/forum?id=iulMde3dP1},
}
Figure