Abstract
Breakthroughs in unsupervised domain adaptation (uDA) can help in adapting models from a label-rich source domain to unlabeled target domains. Despite these advancements, there is a lack of research on how uDA algorithms, particularly those based on adversarial learning, can work in distributed settings. In real-world applications, target domains are often distributed across thousands of devices, and existing adversarial uDA algorithms -- which are centralized in nature -- cannot be applied in these settings. To solve this important problem, we introduce FRuDA: an end-to-end framework for distributed adversarial uDA. Through a careful analysis of the uDA literature, we identify the design goals for a distributed uDA system and propose two novel algorithms to increase adaptation accuracy and training efficiency of adversarial uDA in distributed settings. Our evaluation of FRuDA with five image and speech datasets show that it can boost target domain accuracy by up to 50% and improve the training efficiency of adversarial uDA by at least 11 times.
Introduction
Unsupervised Domain Adaptation (uDA) is essential for deploying machine learning models in diverse real-world environments. It allows a model trained on a labeled source domain to adapt to an unlabeled target domain where data distributions may differ. However, standard uDA assumes centralized data access. In practice, data is often distributed across edge devices (e.g., smartphones, IoT sensors) and cannot be centralized due to privacy regulations or bandwidth constraints.
Existing distributed learning paradigms like Federated Learning primarily address supervised tasks. Applying adversarial uDA—where a feature extractor and a domain discriminator compete—in a distributed, privacy-preserving manner presents significant challenges in communication efficiency and adaptation accuracy. We propose FRuDA (Framework for Realistic uDA) to bridge this gap.
The FRuDA Framework
FRuDA introduces two key algorithmic components to enable efficient distributed adversarial adaptation:
- Optimal Collaborator Selection (OCS): Rather than always adapting from the original source domain, OCS identifies the best "collaborator" node (either the source or a previously adapted target) for a new target domain. It uses a theoretical bound based on the collaborator's internal error and the Wasserstein distance between domains, which can be computed without sharing raw data.
- Discriminator-based Lazy Synchronization (DILS): To minimize communication overhead, DILS keeps feature extractors local and only synchronizes domain discriminator gradients. By employing "lazy" synchronization—exchanging gradients every $p$ steps—it significantly reduces network traffic while maintaining convergence.
Experimental Evaluation
We evaluated FRuDA across five diverse benchmarks, including image (Rotated MNIST, Digits, Office-Caltech, DomainNet) and speech (Mic2Mic) datasets.
The "Better Neighbor" Effect
A core finding is that adapting from a "closer" previously adapted target domain often yields better results than adapting directly from the source. This is particularly evident in the Rotated MNIST experiment. When a 90° rotated domain adapts directly from the 0° source, accuracy is only 16.7%. However, by using OCS to select the 60° domain as a collaborator, accuracy jumps to 92.6%.
Performance and Efficiency
- Accuracy Boost: FRuDA improves target domain accuracy by up to 50% compared to source-only baselines.
- Communication Efficiency: The DILS strategy reduces communication overhead by 11x to 12x. For instance, on the DomainNet dataset, communication was reduced from 86 GB to just 1 GB.
- Privacy: By sharing only discriminator gradients and keeping feature extractors local, FRuDA provides inherent protection against gradient leakage attacks.
Conclusion
FRuDA is the first end-to-end framework for distributed adversarial uDA, offering a scalable and privacy-preserving solution for edge adaptation. By leveraging optimal collaborator selection and lazy gradient synchronization, it achieves significant gains in both accuracy and communication efficiency. Future work will focus on handling concurrent domain joins and exploring adaptation in open-set scenarios.
Summary updated