Abstract
We present a new discriminative technique for the multiple-source adaptation (MSA) problem. Unlike previous work, which relies on density estimation for each source domain, our solution only requires conditional probabilities that can easily be accurately estimated from unlabeled data from the source domains. We give a detailed analysis of our new technique, including general guarantees based on Rényi divergences, and learning bounds when conditional Maxent is used for estimating conditional probabilities for a point to belong to a source domain. We show that these guarantees compare favorably to those that can be derived for the generative solution, using kernel density estimation. Our experiments with real-world applications further demonstrate that our new discriminative MSA algorithm outperforms the previous generative solution as well as other domain adaptation baselines.
Introduction
In modern machine learning, we rarely have a perfect dataset for every new task. Instead, we often have several "source" models trained on different but related domains—for example, one model for sentiment analysis of book reviews and another for electronics. When a new "target" domain arrives (such as kitchen appliance reviews), the goal is to combine these existing models into a single, accurate predictor.
This challenge is known as Multiple-Source Adaptation (MSA). A significant hurdle in MSA is that we often lack labeled data for the new target. Furthermore, for privacy or storage reasons, the original labeled data used to train the source models may be inaccessible; we may only have the models themselves and some unlabeled examples from the various domains.
The Problem with Generative Approaches
Previous state-of-the-art solutions, known as Generative MSA (GMSA), attempted to solve this by estimating the "density" of each source domain—essentially asking, "How likely is this specific input to appear in the 'Books' domain?" However, accurate density estimation is notoriously difficult and data-hungry, especially in high-dimensional spaces like text or images. This often leads to poor performance when unlabeled data is scarce.
Methodology: Discriminative MSA (DMSA)
Researchers from Google and NYU have proposed a Discriminative MSA (DMSA) approach. Instead of trying to model the entire statistical distribution of each source (the generative approach), they train a classifier to predict which source domain a point belongs to (the discriminative approach).
Key Components
- Conditional Probabilities: The method uses a domain classifier (such as conditional Maximum Entropy, or Maxent) to estimate the probability that a given input $x$ belongs to source $k$.
- Distribution-Weighted Combination: The final prediction is a weighted sum of the source models' predictions. Crucially, these weights are dynamic: they change for every input based on the domain classifier's output.
- DC-Programming: To find the optimal global parameters for this combination, the authors derived a new optimization algorithm based on "Difference-of-Convex" (DC) programming, which is computationally efficient.
- Theoretical Guarantees: The authors provide new learning bounds using Rényi divergences, proving mathematically that their discriminative approach is more sample-efficient than the generative one.
Experimental Evaluation
The researchers evaluated DMSA across several diverse benchmarks, including Amazon product reviews (Sentiment Analysis), digit recognition (SVHN, MNIST, and USPS), and the UCI Adult dataset. They also used synthetic Gaussian mixtures to highlight exactly where traditional density estimation (KDE) fails compared to their new method.
Key Results
- Superior Accuracy: On sentiment analysis tasks, DMSA achieved a Mean Squared Error (MSE) of 1.59, compared to 1.65 for GMSA and uniform baselines. In digit recognition, DMSA achieved a mean accuracy of 95.4%, surpassing GMSA (94.7%) and joint training (94.6%).
- High Sample Efficiency: DMSA reaches peak performance with significantly less data. In digit experiments, DMSA reached near-maximum accuracy with only 20-40 samples per domain, whereas the generative approach required hundreds of samples to catch up.
- The Ensemble Bonus: Interestingly, DMSA often outperforms the original source model even when tested on that source's own domain. By "consulting" other models when an input is ambiguous, the system acts as a robust ensemble.
Conclusion and Future Work
The core insight of this research is that classification is fundamentally easier than modeling. It is much simpler to distinguish between two domains (e.g., "Is this a picture of a street sign or a handwritten note?") than it is to perfectly model the statistical distribution of every possible street sign.
Beyond performance, DMSA is privacy-friendly, as it only requires pre-trained predictors and unlabeled data. Future work may apply this robust model-combining approach to algorithmic fairness, ensuring that under-represented groups—who might be treated as "minority domains"—are better served by combining specialized models rather than relying on a single global model.
Summary updated