FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning

Wei Chen, Kartikeya Bhardwaj, Radu Marculescu

Abstract

In this paper, we identify a new phenomenon called activation-divergence which occurs in Federated Learning (FL) due to data heterogeneity (i.e., data being non-IID) across multiple users. Specifically, we argue that the activation vectors in FL can diverge, even if subsets of users share a few common classes with data residing on different devices. To address the activation-divergence issue, we introduce a prior based on the principle of maximum entropy; this prior assumes minimal information about the per-device activation vectors and aims at making the activation vectors of same classes as similar as possible across multiple devices. Our results show that, for both IID and non-IID settings, our proposed approach results in better accuracy (due to the significantly more similar activation vectors across multiple devices), and is more communication-efficient than state-of-the-art approaches in FL. Finally, we illustrate the effectiveness of our approach on a few common benchmarks and two large medical datasets.

Introduction: The Challenge of Data Heterogeneity

Federated Learning (FL) enables training AI models across distributed edge devices without centralizing raw data, preserving privacy. However, a significant hurdle is data heterogeneity. Since local datasets are often non-IID (not identically and independently distributed), local models naturally drift apart during training.

The authors identify a specific manifestation of this drift: activation divergence. Even when different devices share common classes (e.g., "cats"), the internal mathematical representations—activation vectors—can diverge significantly due to the influence of other local data. When a central server aggregates these divergent models, the resulting global model often suffers in performance. FedMAX addresses this by focusing on aligning internal representations rather than just model weights.

The FedMAX Method

FedMAX introduces a regularization technique based on the principle of maximum entropy. While standard FL algorithms like FedAvg focus solely on local classification accuracy, and variants like FedProx constrain local weights to stay near global weights, FedMAX constrains the activation vectors.

Overview of the FedMAX framework and model architecture showing the inclusion of the KL-divergence term in the objective function.

By maximizing the entropy of activation vectors (or minimizing the KL-divergence between local activations and a uniform distribution), FedMAX prevents models from becoming overly specialized to the quirks of local data. This "minimal information" prior encourages the model to learn more universal features that are easier to reconcile during global aggregation.

Experimental Setup and Datasets

The effectiveness of FedMAX was evaluated across five diverse datasets: FEMNIST (handwritten characters), CIFAR-10 & CIFAR-100 (object recognition), and medical datasets APTOS (retina images) and Chest X-ray (pneumonia classification). To simulate realistic non-IID conditions, datasets were partitioned such that each device only observed a small subset of total classes.

Key Results and Insights

FedMAX consistently outperformed state-of-the-art methods like FedAvg and FedProx in both accuracy and efficiency.

Mitigating Divergence

Quantitative analysis shows that FedMAX significantly reduces the divergence of activation vectors compared to FedAvg. This stability allows for more local training epochs without the risk of excessive model drift.

Mean KL-divergence across communication rounds for synthetic datasets, demonstrating FedMAX's superior ability to maintain consistent activations.

Performance Gains

  • Accuracy Boost: On non-IID CIFAR-10, FedMAX achieved a ~5.8% accuracy improvement over baselines.
  • Communication Efficiency: FedMAX reached target accuracy levels up to 5x faster than competing methods. For instance, FedMAX achieved in 600 rounds what FedAvg required 3,000 rounds to reach.
  • Robustness: Unlike simple $L^2$ regularization, which can lead to underfitting, the maximum entropy approach proved robust across various hyper-parameter settings.

Conclusion and Future Work

FedMAX demonstrates that aligning the way models "perceive" data through activation regularization is a powerful tool for overcoming the challenges of non-IID data in Federated Learning. While its impact is most significant in multi-class scenarios where divergence is high, it provides a stable and communication-efficient framework for distributed training. Future research will explore its application in Natural Language Processing (NLP), where linguistic heterogeneity presents a similar challenge for global model convergence.

Summary updated

Abstract

In this paper, we identify a new phenomenon called activation-divergence which occurs in Federated Learning (FL) due to data heterogeneity (i.e., data being non-IID) across multiple users. Specifically, we argue that the activation vectors in FL can diverge, even if subsets of users share a few common classes with data residing on different devices. To address the activation-divergence issue, we introduce a prior based on the principle of maximum entropy; this prior assumes minimal information about the per-device activation vectors and aims at making the activation vectors of same classes as similar as possible across multiple devices. Our results show that, for both IID and non-IID settings, our proposed approach results in better accuracy (due to the significantly more similar activation vectors across multiple devices), and is more communication-efficient than state-of-the-art approaches in FL. Finally, we illustrate the effectiveness of our approach on a few common benchmarks and two large medical datasets.
BibTeX
@inproceedings{chen2020fedmax,
  author = {Wei Chen and Kartikeya Bhardwaj and Radu Marculescu},
  title = {{FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning}},
  booktitle = {ECML/PKDD},
  year = {2020},
  doi = {10.1007/978-3-030-67661-2\_21},
}
Figure