TL;DR
This paper introduces a novel Federated Learning dataset designed to mimic real-world scenarios with domain shift across clients, and proposes two new algorithms, Fed-Cyclic and Fed-Star. These algorithms aim to improve convergence and accuracy in the presence of statistical heterogeneity, outperforming existing baselines.
What Problem Does This Paper Address?
Traditional machine learning often requires centralizing data, which raises privacy concerns and incurs significant costs for data storage and processing. Federated Learning (FL) offers a solution by enabling collaborative model training without sharing raw data. However, existing FL methods like FedAvg struggle with "statistical heterogeneity," where data distributions vary significantly across clients (non-IID data). This can lead to slow convergence, suboptimal performance, and models that don't generalize well to individual clients' data. The paper addresses the gap of evaluating FL in a more realistic commercial setting, where data sources are inherently diverse and exhibit domain shift, and proposes algorithms to tackle the resulting challenges.
Methods
The paper introduces two novel FL algorithms:
- Fed-Cyclic: This algorithm aims to reduce communication bottlenecks. Instead of all clients sending updates to a central server, clients pass their updated model weights cyclically to the next client in a predefined order. This allows for a global model to be updated without direct server involvement in computation, though the server can still facilitate parameter passing to maintain anonymity.
- Fed-Star: This algorithm addresses statistical heterogeneity more directly. Clients first train their local models in parallel. Then, they engage in a "pre-aggregation" step where each client receives weights from all other clients. These weights are then combined using an accuracy-based weighting scheme, prioritizing outlier features from other clients while retaining generalized features. This pre-aggregated model is then used to reinitialize the local model for the next training phase. This process is structured in "periods" within a "round," with final aggregation happening at the central server after a set number of periods.
Datasets
- Newly Introduced Dataset: The authors created a new dataset of 23,326 images collected from 8 different commercial image-hosting websites. Each website serves as a distinct "client" in the FL setting, simulating real-world domain shift. The dataset is classified into 31 categories, inspired by the Office-31 dataset.
- Office-31 Dataset: Mentioned as inspiration for the new dataset's categories.
Key Insights and Ideas
- Real-world FL is Heterogeneous: The paper emphasizes that real-world FL scenarios are rarely IID. Commercial image sources naturally have different distributions, making domain shift a critical factor.
- Fed-Cyclic's Communication Efficiency: By enabling a cyclic flow of weights, Fed-Cyclic reduces reliance on the central server for computation, potentially speeding up training and reducing network congestion.
- Fed-Star's Focus on Outlier Features: The pre-aggregation mechanism in Fed-Star, weighted by accuracy, is designed to help clients learn from the unique "outlier" features of other clients, leading to more personalized and robust models.
- Personalization Matters: The results suggest that FedAvg and RingFed provide less personalized models compared to Fed-Cyclic and Fed-Star, which is crucial for clients with significantly different data.
Key Results
- Global Evaluation: Both Fed-Cyclic and Fed-Star consistently outperformed FedAvg and RingFed across various metrics (accuracy, weighted F1, macro F1). Fed-Star achieved the highest accuracy (91.72% in one experiment).
- Local Evaluation: Fed-Star demonstrated superior performance on individual clients, outperforming local models trained with traditional ML and baselines in most cases. Fed-Cyclic also showed strong performance, often coming in second.
- Convergence: Fed-Star required fewer global rounds (50) to converge compared to FedAvg (250) on their proposed dataset, indicating faster training.
- Learning Rate Sensitivity: Experiments showed that Fed-Star achieved its highest accuracy (92.77%) with a learning rate of 1e-3, while other algorithms had varying optimal learning rates.
Limitations and Future Work
- Communication Intensity of Fed-Star: The paper acknowledges that Fed-Star is communication-intensive due to the pre-aggregation step where each client interacts with every other client. However, they argue that the overall communication overhead with the global server is reduced.
- Future Work: The authors suggest that their proposed algorithms are well-suited for scenarios with statistical heterogeneity and personalization needs, implying further exploration in these areas. The new dataset also provides a valuable benchmark for future FL research in commercial image classification.
Summary updated