Correlated Adversarial Joint Discrepancy Adaptation Network

Youshan Zhang, Brian D. Davison

TL;DR

The Correlated Adversarial Joint Discrepancy Adaptation Network (CAJNet) is a novel framework for unsupervised domain adaptation that aligns both marginal and conditional distributions by learning "joint features." Its standout feature is the introduction of a top-$\mathcal{K}$ correlated label, which provides a principled way to tune hyperparameters and enforce local consistency without "cheating" by using target domain labels.

What Problem Does This Paper Address?

Machine learning models often fail when the data they encounter in the real world (the target domain) differs from their training data (the source domain)—a phenomenon known as domain shift. Unsupervised Domain Adaptation (UDA) aims to fix this by transferring knowledge from a labeled source to an unlabeled target.

However, the authors identify two major gaps in current UDA research:

  1. Feature Tunnel Vision: Most existing methods focus solely on aligning marginal distributions (the general look of the data) while ignoring conditional distributions (how features relate to specific class labels).
  2. The "Tuning Trap": In a true UDA scenario, we have no target labels. Yet, many researchers tune their model hyperparameters using target labels to report better results. This violates the fundamental premise of UDA and makes these models less useful in actual unlabeled environments.

Methods

CAJNet tackles these issues through a multi-pronged architecture designed to minimize the "joint discrepancy" between domains.

The architecture of the CAJNet model. It extracts marginal features, computes conditional features, and concatenates them into "joint features" for a shared encoder.

1. Joint Feature Learning

Instead of just looking at raw features $\Phi(X)$, CAJNet computes conditional features $\Phi(X)^C$ based on class probabilities. By concatenating these into joint features, the model can align both the "what" (marginal) and the "which class" (conditional) aspects of the data simultaneously.

2. The Top-$\mathcal{K}$ Correlated Label ($\mathcal{K}$-label)

This is the paper's most innovative contribution. The authors developed a probabilistic mechanism to identify the $\mathcal{K}$ most similar neighbors for any given target sample.

  • Local Consistency: It enforces a loss ($L_\mathcal{K}$) that ensures a sample's prediction is consistent with its neighbors.
  • Honest Tuning: Crucially, the $\mathcal{K}$-label acts as a proxy for accuracy, allowing researchers to tune hyperparameters (like the learning rate or adaptation factor) using only target domain information.

3. Four-Component Loss Function

The model optimizes a comprehensive objective function:

  • $L_S$ (Classification Loss): Ensures high accuracy on the labeled source data.
  • $L_A$ (Adversarial Loss): Uses a GAN-style discriminator to make features domain-invariant.
  • $L_\mathcal{K}$ (Top-$\mathcal{K}$ Correlated Loss): Enforces local neighborhood consistency in the target domain.
  • $L_{DA}$ (Dynamic Distribution Alignment): Dynamically balances the importance of marginal vs. conditional alignment.

Datasets

The authors evaluated CAJNet on three standard UDA benchmarks:

  • Office + Caltech-10: 10 classes across 4 domains (Amazon, Webcam, DSLR, Caltech).
  • Office-31: 31 classes across 3 domains (Amazon, Webcam, DSLR).
  • Office-Home: A significantly more challenging dataset with 65 classes across 4 visually dissimilar domains (Art, Clipart, Product, Real-World).

Key Insights and Ideas

  • Joint Alignment is Essential: The ablation studies show that aligning both marginal and conditional distributions is superior to aligning either one in isolation. Marginal features carry more raw information, but conditional features are vital for resolving class-level ambiguities.
  • Self-Supervision for Hyperparameters: The $\mathcal{K}$-label provides a path forward for UDA research to move away from using target labels for tuning, making the field more rigorous and the models more deployable.
  • The Power of $\mathcal{K}=3$: Through sensitivity analysis, the authors found that $K=3$ is the "sweet spot" for local consistency. As shown in their analysis, increasing $K$ beyond 4 introduces noise, as neighbors further away are less likely to share the same class.
Effect of different K values on LK loss. The loss is minimized at K=3 across different tasks.

Key Results

CAJNet achieved state-of-the-art (SOTA) performance across all three benchmarks, with particularly impressive gains on the most difficult tasks.

  • Office-Home: Achieved an average accuracy of 75.8%, significantly outperforming previous SOTA methods like SymNets (67.6%) and MEDA (68.3%).
  • Office-31: Reached 91.6% average accuracy.
  • Office + Caltech-10: Reached 97.8% average accuracy.

The ablation study (Table IV) confirms that the full CAJNet model, utilizing all four loss components, consistently performs better than any stripped-down version.

Ablation study on Office-31 dataset showing the contribution of each loss component.

Limitations and Future Work

While CAJNet is robust, the authors acknowledge it isn't a "silver bullet." In a few specific sub-tasks (like $W \to C$ in Office+Caltech), it was slightly outperformed by existing methods. Additionally, while the $\mathcal{K}$-label helps with tuning, $\mathcal{K}$ itself remains a hyperparameter that needs to be set, though it appears stable at $\mathcal{K}=3$ across different datasets. Future work might explore more complex ways to dynamically select $\mathcal{K}$ or apply this joint alignment strategy to even larger, more diverse datasets.

Summary updated

Abstract

Domain adaptation aims to mitigate the domain shift problem when transferring knowledge from one domain into another similar but different domain. However, most existing works rely on extracting marginal features without considering class labels. Moreover, some methods name their model as so-called unsupervised domain adaptation while tuning the parameters using the target domain label. To address these issues, we propose a novel approach called correlated adversarial joint discrepancy adaptation network (CAJNet), which minimizes the joint discrepancy of two domains and achieves competitive performance with tuning parameters using the correlated label. By training the joint features, we can align the marginal and conditional distributions between the two domains. In addition, we introduce a probability-based top-$\mathcal{K}$ correlated label ($\mathcal{K}$-label), which is a powerful indicator of the target domain and effective metric to tune parameters to aid predictions. Extensive experiments on benchmark datasets demonstrate significant improvements in classification accuracy over the state of the art.
BibTeX
@inproceedings{zhang2021correlated,
  author = {Youshan Zhang and Brian D. Davison},
  title = {{Correlated Adversarial Joint Discrepancy Adaptation Network}},
  booktitle = {Cbmi},
  year = {2021},
  doi = {10.1109/cbmi50038.2021.9461907},
}
Figure