Multiple-Source Domain Adaptation via Coordinated Domain Encoders and Paired Classifiers

Payam KarisaniAAAI 2022 · DOI

In the field of machine learning, we often encounter scenarios where labeled data is available from several "source" environments (e.g., news articles or medical journals), but we need to classify data in a "target" environment (e.g., informal tweets) where no labels exist. This paper, presented at AAAI-22, introduces a novel framework called CEPC (Coordinated Encoders and Paired Classifiers) to address the challenges of Multiple-Source Domain Adaptation (MSDA).

The Problem: Negative Transfer and Gradient Domination

While having more data from multiple sources sounds beneficial, it often leads to "negative transfer," where adding source data actually degrades performance. The authors identify two primary causes:

  1. Gradient Domination: If one source domain requires significantly larger updates to align with the target than another, a single shared encoder will be dominated by the "noisier" domain's gradients, resulting in poor representations for the others.
  2. Loss of Local Expertise: Different source classifiers may be "experts" in specific regions of the target feature space. Standard models often average these classifiers, which washes out this specialized knowledge.

The CEPC Framework

The authors propose two primary innovations to mitigate these issues, as illustrated in the model architecture below.

Figure 1: The CEPC architecture illustrating the flow from multiple source and target documents through coordinated encoders to paired classifiers and their respective losses.

1. Coordinated Domain Encoders

Instead of forcing every source domain to share a single encoder or giving every domain its own, CEPC groups domains based on similar "update rates" or scaling factors. By estimating a "pseudo-optimal" scaling factor ($\lambda$) for each domain, the model ensures that no single domain's gradients overwhelm the shared learning process.

2. Paired Classifiers via Heuristics

CEPC identifies which source classifier is most likely to be correct for a specific unlabeled target document using two metrics:

  • Transformation Cost: The distance between source and target covariance matrices, measuring how much a document's representation had to change to fit the source space.
  • Classifier Capacity: A density ratio estimation determining how much source data actually resembles the target document.

Once a "reliable" classifier is identified for a document, it acts as a teacher. Through a KL-divergence loss, it guides the "weaker" classifiers to adjust their class boundaries for that specific region of the data.

Experimental Results and Analysis

CEPC was evaluated on two challenging Twitter datasets: Illness (e.g., Cancer, Diabetes) and Incident (e.g., Flood, Earthquake). These datasets are difficult due to short, informal text and highly imbalanced classes. CEPC emerged as the top-performing model, achieving an average F1-score of 0.742 on the Incident dataset, significantly higher than the next best baseline (JAN-c at 0.702).

Figure 3: Sensitivity analysis showing (a, b) the superiority of CEPC's dynamic coordination over fixed scaling factors (λ) and (c) the impact of the weighting parameter α on F1-score.

Sensitivity analysis confirms that the dynamic coordination of encoders is superior to using fixed scaling factors. As shown in Figure 3, the CEPC approach (represented by the dashed line) consistently maintains higher F1-scores than models using a fixed $\lambda$. Additionally, the performance peaks at a specific $\alpha$ value, validating that the heuristic-based guidance between paired classifiers is a critical driver of the model's success.

Conclusion

CEPC demonstrates that negative transfer is often a gradient management issue and that classifier reliability is local rather than global. By dynamically coordinating encoders and leveraging document-level expertise, CEPC provides a robust framework for unsupervised domain adaptation in complex, real-world scenarios.

Summary updated

BibTeX
@inproceedings{karisani2022multiple,
  author = {Payam Karisani},
  title = {{Multiple-Source Domain Adaptation via Coordinated Domain Encoders and Paired Classifiers}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i7.20668},
}
Figure