Towards Addressing Label Skews in One-Shot Federated Learning

Yiqun Diao, Qinbin Li, Bingsheng HeICLR 2023 · OpenReview

Federated Learning (FL) has emerged as a privacy-preserving paradigm that allows distributed clients to collaboratively train a global model without sharing their raw data. However, traditional FL often requires hundreds of communication rounds between the central server and clients, incurring high communication costs and potential privacy vulnerabilities. "One-shot" FL addresses this by aiming to train a global model in a single communication round.

Despite its promise, one-shot FL struggles significantly when faced with Non-IID data, particularly label skew. In real-world scenarios—such as regional hospitals encountering different subsets of diseases—clients often possess data for only a few classes. If a local model trained exclusively on "cats" is presented with a "dog," it will confidently misclassify the dog as a cat. When a central server aggregates these highly biased models using traditional voting mechanisms, the resulting global model performs poorly because every client confidently misclassifies unseen classes.

To overcome this, researchers have introduced FedOV (Federated learning by Open-set Voting), a novel framework that shifts the paradigm from "close-set" classification to "open-set" recognition, enabling effective one-shot FL even under extreme label skew.

The FedOV Approach: Embracing the "Unknown"

The core innovation of FedOV is teaching local models to admit when they do not know the answer. Instead of forcing a model to select from its limited pool of seen classes, FedOV trains each client as an $(N+1)$-class classifier, where the additional class represents "unknown" data.

Comparison of traditional close-set voting and the proposed open-set voting framework.

During inference, the server aggregates the probabilities for the known classes across all local models but ignores the "unknown" channel. This open-set voting mechanism allows biased models to effectively "abstain" from voting on data they don't recognize, filtering out the noise of confident misclassifications.

To train this "unknown" class without accessing other clients' data, FedOV employs two key techniques:

  1. Data Destruction (DD): Unlike standard data augmentation designed to preserve semantic meaning, DD intentionally corrupts key features using intense operations like RandomSwap and RandomRotation. These heavily corrupted images serve as local proxies for the "unknown" world.
  2. Adversarial Outlier Enhancement (AOE): To establish precise decision boundaries, FedOV uses an adversarial optimization approach. It refines the "destroyed" outliers to resemble the real training data as closely as possible without actually crossing into the known class distribution. This forces the model to draw tight, accurate boundaries around its known classes.
t-SNE visualizations demonstrating how FedOV tightens decision boundaries and pushes unseen classes into the outlier space compared to baseline methods.

Key Results and Insights

The researchers evaluated FedOV across four standard computer vision benchmarks (MNIST, Fashion-MNIST, SVHN, and CIFAR-10), simulating label skew using Dirichlet distributions and fixed class counts.

The results demonstrate massive accuracy gains. In extreme skew settings where each client only has data for a single class, FedOV achieved 40.0% accuracy on CIFAR-10. In stark contrast, traditional close-set voting managed only 10.2%. FedOV consistently outperformed state-of-the-art one-shot methods like FedKT and FedDF across various levels of data imbalance. Furthermore, the method scales robustly; while many FL algorithms degrade under high fragmentation, FedOV maintained strong performance even when tested with up to 80 clients.

Beyond one-shot performance, FedOV also serves as an excellent "warm start" for multi-round FL. Starting a traditional algorithm like FedAvg with a FedOV-initialized model leads to significantly faster convergence and a higher baseline accuracy.

Limitations and Future Directions

While FedOV presents a major leap forward for one-shot FL, it has limitations. The current Data Destruction techniques are vision-centric, relying on image-specific transformations. Adapting this framework for tabular data or Natural Language Processing (NLP) will require novel corruption strategies. Additionally, the diversity of the "unknown" class is bounded by the chosen transformations; future work could explore generative approaches to create richer outlier distributions. Finally, calibrating the probability scales of the "unknown" class across different clients could further refine the voting accuracy, paving the way for even more robust decentralized learning systems.

Summary updated

Abstract

Federated learning (FL) has been a popular research area, where multiple clients collaboratively train a model without sharing their local raw data. Among existing FL solutions, one-shot FL is a promising and challenging direction, where the clients conduct FL training with a single communication round. However, while label skew is a common real-world scenario where some clients may have few or no data of some classes, existing one-shot FL approaches that conduct voting on the local models are not able to produce effective global models. Due to the limited number of classes in each party, the local models misclassify the data from unseen classes into seen classes, which leads to very ineffective global models from voting. To address the label skew issue in one-shot FL, we propose a novel approach named FedOV which generates diverse outliers and introduces them as an additional unknown class in local training to improve the voting performance. Specifically, based on open-set recognition, we propose novel outlier generation approaches by corrupting the original features and further develop adversarial learning to enhance the outliers. Our extensive experiments show that FedOV can significantly improve the test accuracy compared to state-of-the-art approaches in various label skew settings.
BibTeX
@inproceedings{diao2023towards,
  author = {Yiqun Diao and Qinbin Li and Bingsheng He},
  title = {{Towards Addressing Label Skews in One-Shot Federated Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=rzrqh85f4Sc},
}
Figure