Abstract
We present a new discriminative technique for the multiple-source adaptation (MSA) problem. Unlike previous work, which relies on density estimation for each source domain, our solution only requires conditional probabilities that can easily be accurately estimated from unlabeled data from the source domains. We give a detailed analysis of our new technique, including general guarantees based on Rényi divergences, and learning bounds when conditional Maxent is used for estimating conditional probabilities for a point to belong to a source domain. We show that these guarantees compare favorably to those that can be derived for the generative solution, using kernel density estimation. Our experiments with real-world applications further demonstrate that our new discriminative MSA algorithm outperforms the previous generative solution as well as other domain adaptation baselines.
Introduction
In many real-world machine learning applications, we encounter a "data desert" where labeled data for a specific target domain is unavailable. However, we often have access to multiple models trained on related source domains. For instance, a sentiment classifier for book reviews might be available alongside one for electronics, but we need to classify reviews for kitchen appliances. This scenario defines the Multiple-Source Adaptation (MSA) problem: how to intelligently combine existing "expert" models to perform accurately on a new, unseen target domain.
Traditionally, this has been approached through generative methods that require complex density estimation. This paper proposes a shift toward a discriminative approach, leveraging domain classifiers to achieve superior performance with greater sample efficiency.
The Multiple-Source Adaptation Problem
The standard approach to MSA assumes the target distribution is a mixture of several source distributions. The previous state-of-the-art, Generative MSA (GMSA), relies on estimating the probability density of each source. It asks: "How likely is it that this specific input $x$ appears in this specific source domain?"
However, high-dimensional density estimation is notoriously difficult and data-intensive. Small errors in density estimation can lead to significant failures in the combined model's predictions. If the estimated density for a relevant source is slightly off, the model may incorrectly discount that source's expertise.
Discriminative MSA (DMSA)
The proposed Discriminative MSA (DMSA) algorithm simplifies the task. Instead of full density estimation, it uses domain classifiers to determine the conditional probability that a given point $x$ belongs to a particular source domain. This shifts the problem from asking "how likely is $x$?" to "given $x$, which domain is it from?"
Key Technical Highlights
- Domain Classifiers: The method uses conditional probabilities (e.g., from Logistic Regression or Maxent models) to weight the source predictors. This is a significantly easier learning task than density estimation.
- Theoretical Guarantees: The authors provide learning bounds using Rényi divergences to measure the distance between distributions. They establish a "triangle inequality" for these divergences, which helps bound the target error.
- Optimization via DC-Programming: Finding the optimal weights for combining source models is formulated as a Difference-of-Convex (DC) programming problem, solvable through an efficient iterative algorithm.
Experimental Evaluation
The researchers validated DMSA across several diverse benchmarks, demonstrating its robustness and efficiency:
- Sentiment Analysis: Using Amazon product reviews (Books, DVD, Electronics, Kitchen), DMSA consistently outperformed GMSA and uniform blending across various target mixtures.
- Digit Recognition: On a task combining SVHN, MNIST, and USPS datasets, DMSA achieved a mean accuracy of 95.4%, surpassing GMSA (94.7%) and joint training (94.6%).
- UCI Adult & Synthetic Data: Experiments on income prediction and synthetic Gaussian mixtures further confirmed that DMSA is more robust than generative methods, especially when density estimation produces "bumps" in the wrong places.
- Sample Efficiency: DMSA requires significantly less unlabeled data. In digit recognition experiments, DMSA reached near-peak accuracy with only 40 samples per domain, while GMSA required over 80 to reach comparable levels.
Conclusion
DMSA represents a significant advancement in multiple-source adaptation by adhering to the principle of solving the simplest possible intermediate task. By replacing generative density estimation with discriminative domain classification, the algorithm becomes more robust to data scarcity and achieves higher accuracy. While currently limited to target domains within the convex hull of the sources, future work may extend these guarantees to more complex, non-mixture target distributions.
Summary updated