A Discriminative Technique for Multiple-Source Adaptation

Corinna Cortes, Mehryar Mohri, Ananda Theertha Suresh, Ningshan Zhang

Introduction: The Multiple-Source Adaptation Problem

In many real-world machine learning scenarios, we possess accurate models trained on specific "source" domains—such as sentiment analysis models fine-tuned for books or electronics. However, we frequently need to apply these models to a new "target" domain (e.g., a mix of various product categories) for which we have no labeled data. This challenge is known as the Multiple-Source Adaptation (MSA) problem.

Previous state-of-the-art solutions relied on a generative approach. They attempted to estimate the probability density of each source domain to determine how to weight the different predictors for the target domain. Unfortunately, density estimation is notoriously difficult, especially in high-dimensional spaces due to the "curse of dimensionality." If the density estimates are even slightly inaccurate, the final combined predictor can perform poorly.

To overcome this, the authors propose Discriminative Multiple-Source Adaptation (DMSA), a novel method that shifts from generative density estimation to a discriminative approach, achieving higher accuracy with significantly less data and stronger theoretical guarantees.

From Generative to Discriminative Modeling

The core innovation of DMSA is the transition from generative to discriminative modeling. Instead of asking the generative question, "What is the probability of seeing this specific data point in Domain A?" ($P(x|k)$), the authors ask the discriminative question, "Given this data point, what is the probability it belongs to Domain A?" ($P(k|x)$).

Key components of this approach include:

  • Conditional Probabilities: The model estimates the probability that a point $x$ belongs to source domain $k$. This frames the problem as a standard classification task, which is generally much easier and more robust than high-dimensional density estimation.
  • Conditional Maxent: The authors utilize conditional maximum entropy (multinomial logistic regression) to estimate these domain-membership probabilities directly from unlabeled data.
  • DC-Programming: To find the optimal mixture weights for combining the source predictors, the paper introduces an efficient optimization algorithm based on Difference-of-Convex (DC) programming.

Theoretical Insights: The Total Sample Advantage

A major breakthrough in the paper is the difference in sample complexity between the two approaches. The paper provides new learning bounds based on Rényi divergences, proving that DMSA is theoretically more "sample-efficient" than generative methods.

The error bound for traditional generative approaches depends on the number of samples per domain. In stark contrast, DMSA’s bound depends on the total number of samples across all domains. This "total sample advantage" makes DMSA significantly more powerful as the number of source domains increases. Furthermore, generative methods like Kernel Density Estimation (KDE) are extremely sensitive to bandwidth parameters; DMSA avoids this fragility entirely by treating domain membership as a classification problem.

Empirical Results

The authors evaluated DMSA across a diverse set of benchmarks, including Sentiment Analysis (Amazon reviews), Digit Recognition (SVHN, MNIST, USPS), the Office Dataset, the Adult Dataset, and a custom Synthetic Dataset designed to highlight where traditional density estimation fails.

DMSA consistently outperformed the previous generative state-of-the-art (GMSA) and other baselines across all experiments.

Sample Efficiency

In tasks like digit recognition, DMSA demonstrated its theoretical sample efficiency in practice, reaching high accuracy much faster than generative baselines as the number of unlabeled samples increased.

Accuracy vs. number of samples demonstrating DMSA's sample efficiency

Broad Success Across Domains

In the sentiment analysis benchmark, DMSA outperformed all baselines on almost every target mixture, often by a statistically significant margin. Even when the target domain was exactly one of the source domains, DMSA frequently outperformed a single model trained exclusively on that domain because it effectively leveraged an ensemble of all available predictors.

Performance results on the Sentiment Analysis dataset

On the synthetic dataset, DMSA achieved nearly 100% accuracy, while the generative approach struggled because its density estimates were inaccurate in regions where domain distributions overlapped.

Limitations and Future Work

While DMSA presents a significant step forward, the authors note a few limitations:

  • Convex Hull Assumption: The theoretical guarantees assume the target distribution lies within (or very close to) the "convex hull" of the source distributions—meaning the target is essentially some mixture of the known sources.
  • Shared Conditionals: The current model assumes that the relationship between features and labels ($P(y|x)$) remains the same across all domains.

Future work could explore relaxing these assumptions, utilizing more complex domain classifiers like deep neural networks, and investigating scenarios where the source domains have significantly different labeling functions.

Summary updated

Abstract

We present a new discriminative technique for the multiple-source adaptation (MSA) problem. Unlike previous work, which relies on density estimation for each source domain, our solution only requir...
BibTeX
@inproceedings{cortes2021discriminative,
  author = {Corinna Cortes and Mehryar Mohri and Ananda Theertha Suresh and Ningshan Zhang},
  title = {{A Discriminative Technique for Multiple-Source Adaptation}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=qalHgelJlb},
}
Figure