Federated Learning (FL) has emerged as a privacy-preserving paradigm that allows distributed clients to collaboratively train a global model without sharing their raw data. However, traditional FL often requires hundreds of communication rounds between the central server and clients, incurring high communication costs and potential privacy vulnerabilities. "One-shot" FL addresses this by aiming to train a global model in a single communication round.
Despite its promise, one-shot FL struggles significantly when faced with Non-IID data, particularly label skew. In real-world scenarios—such as regional hospitals encountering different subsets of diseases—clients often possess data for only a few classes. If a local model trained exclusively on "cats" is presented with a "dog," it will confidently misclassify the dog as a cat. When a central server aggregates these highly biased models using traditional voting mechanisms, the resulting global model performs poorly because every client confidently misclassifies unseen classes.
To overcome this, researchers have introduced FedOV (Federated learning by Open-set Voting), a novel framework that shifts the paradigm from "close-set" classification to "open-set" recognition, enabling effective one-shot FL even under extreme label skew.
The FedOV Approach: Embracing the "Unknown"
The core innovation of FedOV is teaching local models to admit when they do not know the answer. Instead of forcing a model to select from its limited pool of seen classes, FedOV trains each client as an $(N+1)$-class classifier, where the additional class represents "unknown" data.
During inference, the server aggregates the probabilities for the known classes across all local models but ignores the "unknown" channel. This open-set voting mechanism allows biased models to effectively "abstain" from voting on data they don't recognize, filtering out the noise of confident misclassifications.
To train this "unknown" class without accessing other clients' data, FedOV employs two key techniques:
- Data Destruction (DD): Unlike standard data augmentation designed to preserve semantic meaning, DD intentionally corrupts key features using intense operations like
RandomSwapandRandomRotation. These heavily corrupted images serve as local proxies for the "unknown" world. - Adversarial Outlier Enhancement (AOE): To establish precise decision boundaries, FedOV uses an adversarial optimization approach. It refines the "destroyed" outliers to resemble the real training data as closely as possible without actually crossing into the known class distribution. This forces the model to draw tight, accurate boundaries around its known classes.
Key Results and Insights
The researchers evaluated FedOV across four standard computer vision benchmarks (MNIST, Fashion-MNIST, SVHN, and CIFAR-10), simulating label skew using Dirichlet distributions and fixed class counts.
The results demonstrate massive accuracy gains. In extreme skew settings where each client only has data for a single class, FedOV achieved 40.0% accuracy on CIFAR-10. In stark contrast, traditional close-set voting managed only 10.2%. FedOV consistently outperformed state-of-the-art one-shot methods like FedKT and FedDF across various levels of data imbalance. Furthermore, the method scales robustly; while many FL algorithms degrade under high fragmentation, FedOV maintained strong performance even when tested with up to 80 clients.
Beyond one-shot performance, FedOV also serves as an excellent "warm start" for multi-round FL. Starting a traditional algorithm like FedAvg with a FedOV-initialized model leads to significantly faster convergence and a higher baseline accuracy.
Limitations and Future Directions
While FedOV presents a major leap forward for one-shot FL, it has limitations. The current Data Destruction techniques are vision-centric, relying on image-specific transformations. Adapting this framework for tabular data or Natural Language Processing (NLP) will require novel corruption strategies. Additionally, the diversity of the "unknown" class is bounded by the chosen transformations; future work could explore generative approaches to create richer outlier distributions. Finally, calibrating the probability scales of the "unknown" class across different clients could further refine the voting accuracy, paving the way for even more robust decentralized learning systems.
Summary updated