Abstract
Semi-Supervised Domain Adaptation (SSDA) aims to leverage labeled source data and limited labeled target data to improve model performance on a target domain. However, existing methods often suffer from alignment bias and architectural rigidity. This paper introduces Semi-Supervised Transfer Boosting (SS-TrBoosting), a novel framework that enhances pre-trained deep domain adaptation models by incorporating an ensemble of boosting-based fine-tuning blocks. By iteratively applying supervised and self-supervised learners, SS-TrBoosting effectively reduces alignment bias and maximizes the utility of unlabeled target data.
Introduction
Domain shift remains a significant challenge in machine learning, where models trained on a source domain fail to generalize to a related target domain. SSDA addresses this by utilizing a large labeled source dataset, a small set of labeled target samples, and a substantial amount of unlabeled target data. Traditional deep learning approaches focus on learning domain-invariant features, but they often encounter "alignment bias"—where even perfectly aligned features lead to biased classification—and lack the flexibility to easily integrate new labeled target data into existing unsupervised models. The authors propose that instead of solely focusing on feature alignment, significant gains can be made by modifying the classifier itself to be more flexible and robust.
Methodology
SS-TrBoosting shifts the focus from feature extraction to the classifier through an ensemble approach. The framework begins with a pre-trained deep Unsupervised Domain Adaptation (UDA) or SSDA model, freezing its feature extractor to maintain stable representations. It then appends a series of $K$ fine-tuning blocks to the classifier.
The Architecture
Each fine-tuning block consists of two specialized base learners:
- Supervised DA Learner: This component focuses on the labeled source and target data to correct alignment bias. It utilizes a re-weighting strategy inspired by TrAdaBoost to prioritize source samples that align with target labels while discarding contradictory ones, thereby avoiding "negative transfer."
- SSL Learner: This component exploits unlabeled target data through pseudo-labeling and feature-level data augmentation (e.g., adding Gaussian noise). This allows the model to better capture the underlying structure of the target domain.
These learners are integrated using LogitBoost, which is computationally efficient and well-suited for the regression-based training of the base learners. To ensure diversity and speed, each base learner employs a random nonlinear feature mapping, similar to an Extreme Learning Machine, allowing the model to add complexity without the massive overhead of training deep neural networks for every boosting iteration.
Source-Free Extension (SS-SFDA)
For privacy-sensitive applications where source data is unavailable, the authors propose a "virtual source domain generation" method. This synthesizes virtual source features from the weights of the pre-trained classifier, acting as prototypes to guide the boosting process without requiring the original dataset.
Experiments and Results
The framework was rigorously evaluated on three standard benchmarks: DomainNet (large-scale), Office-Home (medium-scale), and Office-31 (small-scale). SS-TrBoosting consistently improved the performance of every baseline model it was tested against, including popular models like MME, APE, and CDAN.
Key quantitative findings include:
- Universal Improvement: On the large-scale DomainNet dataset, SS-TrBoosting achieved an average improvement of 1.5% in 1-shot settings and 0.8% in 3-shot settings across various baselines.
- Source-Free Success: In source-free scenarios, it improved the "SourceOnly" baseline by approximately 3.8% on Office-Home, validating the synthesized data as a viable substitute for real source data.
- Computational Efficiency: Compared to simple ensembles of deep models, SS-TrBoosting is orders of magnitude faster because it only optimizes small linear models on top of fixed features.
Conclusion
SS-TrBoosting provides a robust and efficient way to "supercharge" existing domain adaptation models. By focusing on the classifier and utilizing a boosting-based ensemble of supervised and self-supervised blocks, it overcomes the limitations of traditional feature-alignment methods. While the framework's performance is partially dependent on the quality of the initial pre-trained model, its ability to reduce alignment bias and prevent negative transfer makes it a powerful tool for SSDA. Future work will explore extending this framework to fully unsupervised and extremely low-shot domain adaptation scenarios.
Summary updated