TL;DR
This paper introduces DATS, a novel method for adversarial domain adaptation that specifically tackles "label shift" – a common scenario where class distributions differ between training and testing data. DATS achieves robust performance by estimating target label proportions and adaptively weighting source domains, outperforming existing methods on challenging synthetic and real-world datasets.
What Problem Does This Paper Address?
Machine learning models often struggle when the data they are trained on (source domain) differs from the data they encounter in the real world (target domain). This "domain shift" is a major hurdle for generalization. While adversarial domain adaptation methods have shown promise in learning domain-invariant features, they often implicitly assume that the proportion of samples belonging to each class remains constant across domains. This assumption is frequently violated in practice, especially in fields like behavioral studies where naturally occurring variations lead to significant differences in class distributions, known as "label shift" or "target shift." Existing methods that ignore this shift can lead to suboptimal performance or even failure. This paper aims to bridge this gap by developing a method that explicitly accounts for label shift in unsupervised domain adaptation, even when multiple source domains are available.
Methods
The core of the proposed method, DATS (Domain Adversarial nets for Target Shift), builds upon the Domain Adversarial Neural Network (DANN) framework. DATS introduces two key innovations:
- Label Proportion Estimation and Weighting: DATS estimates the unknown label proportions in the target domain. This is achieved by minimizing the discrepancy between the marginal distributions of features in the source and target domains, after accounting for the estimated target label proportions. This estimated proportion is then used to reweight source domain samples, effectively correcting for the label imbalance during the adversarial training process.
- Multi-Source Domain Weighting: To handle scenarios with multiple source domains, DATS learns a weighting scheme that adaptively upweights source domains most similar to the target domain. This prevents irrelevant or noisy source domains from negatively impacting performance, a common issue in multi-source domain adaptation.
These components are integrated into an end-to-end trainable neural network framework. The method uses distribution matching (specifically, kernel mean matching and f-divergence) to estimate label proportions and a softmax-based scheme to learn domain weights.
Datasets
The paper evaluates DATS on the following datasets:
- Synthetically altered MNIST: MNIST digits '4' and '9' are used, with synthetically manipulated label proportions in the target domain (MNISTM) to simulate label shift.
- Real Datasets:
- ASD Dataset: Electroencephalography (EEG) signals from children undergoing treatment for Autism Spectral Disorder (ASD). The target is the treatment stage (before, 6 months, 12 months post-treatment).
- LFP Dataset: Local Field Potential (LFP) signals from mice, used to predict behavior conditions (home cage, open field, tail suspension).
The experiments employ a "leave-one-subject-out" cross-validation strategy for the real datasets, treating each subject as a separate domain.
Key Insights and Ideas
- Label Shift is a Critical Problem: The paper strongly emphasizes that ignoring label shift in domain adaptation can lead to significant performance degradation, even with powerful adversarial methods.
- Estimating Target Labels is Key: Directly estimating the target domain's label distribution, rather than assuming it's the same as the source, is crucial for robust adaptation.
- Adaptive Source Domain Weighting is Necessary: In multi-source scenarios, not all sources are equally helpful. Learning to prioritize more relevant sources is vital for optimal performance.
- Adversarial Framework is Adaptable: The core adversarial mechanism for learning domain-invariant features can be effectively extended to handle label shift and multi-source complexities.
Key Results
- Synthetic Data: DATS demonstrates robust performance across a wide range of label shift scenarios on the synthetic MNIST dataset, significantly outperforming baselines like DANN and a "weighted source" approach. The estimated target label proportions were found to be highly accurate (within 0.05 difference from ground truth).
- Real Data: On both the ASD and LFP datasets, DATS consistently achieved higher classification accuracy compared to existing methods, including DANN, MDANs, and SyncNet. For instance, on the ASD LFP dataset, DATS achieved 67.2% accuracy, outperforming the next best method (Mean Matching) by 2%.
- Robustness: The proposed method is shown to be robust to large label shifts and performs well even when the number of source domains is large.
Limitations and Future Work
- Computational Cost: While the paper mentions that some optimization-based methods are hindered by matrix inversion for large-scale data, the proposed method's computational efficiency is not explicitly detailed beyond stating it's trained end-to-end.
- Positive Feedback Loop in Domain Weighting: The paper acknowledges a potential positive feedback loop in the domain weighting scheme, which could theoretically focus on irrelevant domains. However, they state that empirically, this has not been an issue and has instead led to performance improvements by narrowing focus to relevant domains.
- Future Work: The paper suggests that the framework can be extended to handle more complex scenarios and that further exploration of distribution matching techniques could yield additional improvements. The authors also note that the theoretical proof of consistency for label proportion estimation is strictly for a single source domain, though it's straightforward to extend to multiple domains.
Summary updated