FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning

Wei Chen, Kartikeya Bhardwaj, Radu Marculescu

TL;DR

FedMAX introduces a novel regularization technique for Federated Learning (FL) based on the principle of maximum entropy. By mitigating "activation divergence"—a phenomenon where different devices represent the same concepts in wildly different ways—FedMAX achieves significantly higher accuracy and up to a 5x reduction in communication costs compared to industry standards like FedAvg and FedProx.

What Problem Does This Paper Address?

Federated Learning allows AI models to be trained across thousands of edge devices (like phones or medical sensors) without ever moving the raw data to a central server. While great for privacy, it faces a massive hurdle: data heterogeneity. Because your photo library looks nothing like mine (non-IID data), the local models we train will naturally drift apart.

The authors identify a specific flavor of this drift called activation divergence. Even if two devices both have images of "cats," the internal mathematical representations (activation vectors) they create for those cats can diverge because the rest of their local data is so different. When a central server tries to average these "divergent" models, the resulting global model often performs poorly. Previous solutions tried to keep model weights similar, but FedMAX argues we should focus on making the internal representations similar.

Methods

The core of FedMAX is a new "prior" or constraint added to the local training process. Instead of just minimizing classification error, FedMAX forces the model to maximize the entropy of its activation vectors (the layer just before the final output).

  • The Standard Way: Most FL algorithms (like FedAvg) just focus on local accuracy.
  • The FedProx Way: Constrains local weights to stay physically close to the global model's weights.
  • The FedMAX Way: Uses the Principle of Maximum Entropy. By maximizing entropy (or, equivalently, minimizing the KL-divergence between local activations and a uniform distribution), the algorithm prevents the model from becoming too "specialized" or biased toward the specific quirks of a single device's data.

Essentially, it encourages the model to find a more universal, stable way to represent features that can be easily reconciled with models from other devices.

Datasets

The researchers evaluated FedMAX across five diverse datasets to prove its versatility:

  1. FEMNIST*: A handwritten character dataset.
  2. CIFAR-10 & CIFAR-100: Standard object recognition benchmarks.
  3. APTOS: A medical dataset for detecting blindness from retina images.
  4. Chest X-ray: A medical dataset for classifying pneumonia.

To simulate real-world "non-IID" conditions, they partitioned these datasets so that each simulated device only saw a tiny subset of the total classes (e.g., a device might only see 2 out of 10 CIFAR-10 labels).

Key Insights and Ideas

  • Activations Matter More Than Weights: While weight divergence is a known issue, activation divergence is a more direct proxy for how a model "perceives" the world. Fixing the perception makes the global aggregation much smoother.
  • Stability over $L^2$: The authors tested simple $L^2$ regularization (which just keeps activation values small), but found it caused underfitting. Maximum entropy provides a "minimal information" prior that is far more robust to hyper-parameter changes.
  • Communication is the Bottleneck: In FL, sending model updates over the air is expensive and slow. FedMAX allows for more "local work" (more training epochs on the device) without the models drifting too far apart, which drastically cuts down the number of times a device needs to talk to the server.

Key Results

FedMAX consistently outperformed the state-of-the-art:

  • Accuracy Boost: On the non-IID CIFAR-10 dataset, FedMAX achieved roughly 5.8% higher accuracy than FedAvg and FedProx.
  • Communication Efficiency: FedMAX reached higher accuracy in just 600 rounds than the other methods reached in 3,000 rounds—a 5x speedup in training time.
  • Visual Proof: Using t-SNE plots (a way to visualize high-dimensional data), the authors showed that FedMAX successfully forced activations of the same class from different devices to cluster together, whereas FedAvg's activations remained scattered and disorganized.

Limitations and Future Work

The authors noted that FedMAX's benefits were less pronounced on the Chest X-ray dataset. They attribute this to the dataset having only two classes; with so few categories, there is less "room" for activations to diverge significantly, making the entropy regularization less impactful.

Looking ahead, the team plans to test FedMAX on even larger datasets with more classes and explore its application in Natural Language Processing (NLP), where data heterogeneity in how people speak or write is a major challenge for federated models.

Summary updated

Abstract

In this paper, we identify a new phenomenon called activation divergence which occurs in Federated Learning (FL) due to data heterogeneity (i.e., data being non-IID) across multiple users. Specifically, we argue that the activation vectors in FL can diverge, even if subsets of users share a few common classes with data residing on different devices. To address the activation-divergence issue, we introduce a prior based on the principle of maximum entropy; this prior assumes minimal information about the per-device activation vectors and aims at making the activation vectors of same classes as similar as possible across multiple devices. Our results show that, for both IID and non-IID settings, our proposed approach results in better accuracy (due to the significantly more similar activation vectors across multiple devices), and is more communication-efficient than state-of-the-art approaches in FL. Finally, we illustrate the effectiveness of our approach on a few common benchmarks and two large medical datasets.
BibTeX
@inproceedings{chen2020fedmax,
  author = {Wei Chen and Kartikeya Bhardwaj and Radu Marculescu},
  title = {{FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning}},
  booktitle = {ECML/PKDD},
  year = {2020},
  doi = {10.1007/978-3-030-67661-2\_21},
}

Loading related content...

Figure