FRuDA: Framework for Distributed Adversarial Domain Adaptation

Shaoduo Gan, Akhil Mathur, Anton Isopoussu, Fahim Kawsar, Nadia Berthouze, Nicholas D. Lane

TL;DR

FRuDA (Framework for Realistic uDA) is the first end-to-end framework designed to perform unsupervised domain adaptation across thousands of distributed, privacy-sensitive devices. By intelligently selecting the best "collaborator" domain and using a "lazy" gradient synchronization strategy, it boosts adaptation accuracy by up to 50% while reducing communication overhead by over 11x.

What Problem Does This Paper Address?

Unsupervised Domain Adaptation (uDA) is a powerful tool for taking a model trained on a labeled source domain (like high-quality studio speech) and making it work on an unlabeled target domain (like noisy speech from a specific smartphone).

Standard uDA algorithms assume all your data is sitting on one big server. In the real world, however, data is often "trapped" on user devices due to privacy regulations or bandwidth constraints. If you want to adapt a COVID-19 cough-detection model to thousands of individual users, you can't just upload their raw medical data to a central hub.

Existing distributed learning methods (like Federated Learning) mostly focus on supervised tasks. Applying adversarial uDA—where a "discriminator" tries to tell domains apart while a "feature extractor" tries to fool it—in a distributed, privacy-preserving, and communication-efficient way remains a massive challenge.

Methods

The authors introduce FRuDA, which consists of two core algorithmic innovations:

  1. Optimal Collaborator Selection (OCS): In a system with many domains, the original source isn't always the best starting point for a new user. OCS uses a novel theoretical bound to estimate which existing domain (either the source or a previously adapted target) will yield the highest accuracy for a new target. It calculates this using the "collaborator's" internal error and the Wasserstein distance between the two domains—a metric that can be computed without ever exchanging raw data.
  2. Discriminator-based Lazy Synchronization (DILS): To make training efficient, DILS splits the adversarial architecture. The feature extractors stay local to each node, while only the gradients of the domain discriminator are exchanged. Crucially, it uses "lazy" synchronization, meaning nodes only talk to each other every $p$ steps. This drastically cuts down on network traffic without sacrificing the model's ability to converge.

Datasets

The researchers evaluated FRuDA across five diverse image and speech benchmarks:

  • Rotated MNIST: Digits rotated at various angles (treating each angle as a domain).
  • Digits: A collection of five distinct digit datasets (MNIST, USPS, SVHN, MNIST-M, and SynNumbers).
  • Office-Caltech: 10 object classes across four domains (Amazon, DSLR, Webcam, Caltech-256).
  • Mic2Mic: A speech keyword detection dataset recorded using four different types of microphones.
  • DomainNet: A massive dataset featuring four domains (Real, QuickDraw, Infograph, Sketch) with 345 classes.

Key Insights and Ideas

  • The "Better Neighbor" Effect: One of the most striking insights is that a new target domain often achieves much higher accuracy by adapting from a previously adapted target rather than the original labeled source. For example, on Rotated MNIST, a 90° domain performs 75% better when it "collaborates" with a 60° domain instead of the 0° source.
  • Privacy through Gradient Sparsity: By only exchanging discriminator gradients and keeping feature extractor weights local, FRuDA is naturally resistant to state-of-the-art "gradient leakage" attacks that try to reconstruct raw images from shared model updates.
  • Efficiency vs. Accuracy Trade-off: The "lazy" synchronization parameter ($p$) acts as a tunable knob. The authors found that syncing every 4 to 10 steps provides a massive boost in training speed with negligible impact on final accuracy.

Key Results

  • Accuracy Boost: FRuDA outperformed standard baselines (like always using the labeled source) in almost all scenarios, increasing target domain accuracy by up to 50%.
  • Communication Efficiency: Compared to strict synchronization methods (like FADA), DILS reduced the amount of data exchanged by 11x to 12x. On the DomainNet dataset, this meant dropping from 86 GB of traffic to just 1 GB.
  • Generalizability: The framework is "algorithm-agnostic," meaning it can support various adversarial uDA loss functions (like DANN, ADDA, or CADA) within its distributed structure.

Limitations and Future Work

While FRuDA is a major step forward, the authors acknowledge a few areas for improvement:

  • Sequential Bottleneck: Currently, the framework processes target domains one by one as they join the system. In a massive deployment, handling "batches" of new domains simultaneously would be more practical.
  • Label Space Assumptions: The paper assumes the source and target domains share the same set of classes (e.g., they are both looking for digits 0-9). Future work will explore "Partial" or "Open-set" adaptation where the classes might differ.
  • Error Propagation: The OCS selection relies on a small local test set on the collaborator node; the authors plan to investigate more sophisticated ways to propagate error estimates across the network.

Summary updated

Abstract

Breakthroughs in unsupervised domain adaptation (uDA) can help in adapting models from a label-rich source domain to unlabeled target domains. Despite these advancements, there is a lack of research on how uDA algorithms, particularly those based on adversarial learning, can work in distributed settings. In real-world applications, target domains are often distributed across thousands of devices, and existing adversarial uDA algorithms – which are centralized in nature – cannot be applied in these settings. To solve this important problem, we introduce FRuDA: an end-to-end framework for distributed adversarial uDA. Through a careful analysis of the uDA literature, we identify the design goals for a distributed uDA system and propose two novel algorithms to increase adaptation accuracy and training efficiency of adversarial uDA in distributed settings. Our evaluation of FRuDA with five image and speech datasets show that it can boost target domain accuracy by up to 50% and improve the training efficiency of adversarial uDA by at least <inline-formula><tex-math notation="LaTeX">$11\times$</tex-math></inline-formula> .
BibTeX
@article{gan2022fruda,
  author = {Shaoduo Gan and Akhil Mathur and Anton Isopoussu and Fahim Kawsar and Nadia Berthouze and Nicholas D. Lane},
  title = {{FRuDA: Framework for Distributed Adversarial Domain Adaptation}},
  journal = {IEEE Trans. Parallel Distributed Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=nxwcTfJHXGHo},
}

Loading related content...

Figure