Abstract
In this paper, we identify a new phenomenon called activation-divergence which occurs in Federated Learning (FL) due to data heterogeneity (i.e., data being non-IID) across multiple users. Specifically, we argue that the activation vectors in FL can diverge, even if subsets of users share a few common classes with data residing on different devices. To address the activation-divergence issue, we introduce a prior based on the principle of maximum entropy; this prior assumes minimal information about the per-device activation vectors and aims at making the activation vectors of same classes as similar as possible across multiple devices. Our results show that, for both IID and non-IID settings, our proposed approach results in better accuracy (due to the significantly more similar activation vectors across multiple devices), and is more communication-efficient than state-of-the-art approaches in FL. Finally, we illustrate the effectiveness of our approach on a few common benchmarks and two large medical datasets.
Introduction: The Challenge of Data Heterogeneity
Federated Learning (FL) enables training AI models across distributed edge devices without centralizing raw data, preserving privacy. However, a significant hurdle is data heterogeneity. Since local datasets are often non-IID (not identically and independently distributed), local models naturally drift apart during training.
The authors identify a specific manifestation of this drift: activation divergence. Even when different devices share common classes (e.g., "cats"), the internal mathematical representations—activation vectors—can diverge significantly due to the influence of other local data. When a central server aggregates these divergent models, the resulting global model often suffers in performance. FedMAX addresses this by focusing on aligning internal representations rather than just model weights.
The FedMAX Method
FedMAX introduces a regularization technique based on the principle of maximum entropy. While standard FL algorithms like FedAvg focus solely on local classification accuracy, and variants like FedProx constrain local weights to stay near global weights, FedMAX constrains the activation vectors.
By maximizing the entropy of activation vectors (or minimizing the KL-divergence between local activations and a uniform distribution), FedMAX prevents models from becoming overly specialized to the quirks of local data. This "minimal information" prior encourages the model to learn more universal features that are easier to reconcile during global aggregation.
Experimental Setup and Datasets
The effectiveness of FedMAX was evaluated across five diverse datasets: FEMNIST (handwritten characters), CIFAR-10 & CIFAR-100 (object recognition), and medical datasets APTOS (retina images) and Chest X-ray (pneumonia classification). To simulate realistic non-IID conditions, datasets were partitioned such that each device only observed a small subset of total classes.
Key Results and Insights
FedMAX consistently outperformed state-of-the-art methods like FedAvg and FedProx in both accuracy and efficiency.
Mitigating Divergence
Quantitative analysis shows that FedMAX significantly reduces the divergence of activation vectors compared to FedAvg. This stability allows for more local training epochs without the risk of excessive model drift.
Performance Gains
- Accuracy Boost: On non-IID CIFAR-10, FedMAX achieved a ~5.8% accuracy improvement over baselines.
- Communication Efficiency: FedMAX reached target accuracy levels up to 5x faster than competing methods. For instance, FedMAX achieved in 600 rounds what FedAvg required 3,000 rounds to reach.
- Robustness: Unlike simple $L^2$ regularization, which can lead to underfitting, the maximum entropy approach proved robust across various hyper-parameter settings.
Conclusion and Future Work
FedMAX demonstrates that aligning the way models "perceive" data through activation regularization is a powerful tool for overcoming the challenges of non-IID data in Federated Learning. While its impact is most significant in multi-class scenarios where divergence is high, it provides a stable and communication-efficient framework for distributed training. Future research will explore its application in Natural Language Processing (NLP), where linguistic heterogeneity presents a similar challenge for global model convergence.
Summary updated