Abstract
Federated Learning (FL) faces significant challenges in real-world deployments due to non-identical data distributions across clients (statistical heterogeneity) and temporal changes in local data (distribution drift). Traditional FL approaches often fail to adapt to these dynamics, leading to suboptimal global models. We propose FedCM, a novel framework that addresses these issues through a multi-stage process: a warm-up phase to capture gradient trajectories, a graph-based clustering algorithm using a new "gradient path" similarity metric, and a dynamic migration strategy based on update direction deviation. Our evaluations on standard benchmarks demonstrate that FedCM significantly outperforms existing state-of-the-art methods in both accuracy and robustness to data drift.
Introduction
Federated Learning enables collaborative model training while preserving data privacy. However, two primary obstacles hinder its effectiveness:
- Statistical Heterogeneity: Diverse client data distributions (e.g., label skew) make a single global model insufficient for all participants.
- Data Distribution Drift: Client data patterns often evolve over time, rendering static clustering or global models obsolete.
Existing clustered federated learning (CFL) methods typically rely on static snapshots of model weights or require a predefined number of clusters, which limits their flexibility in dynamic environments. FedCM overcomes these limitations by analyzing the learning trajectory of clients and allowing for seamless migration between clusters as data drifts.
Methodology: The FedCM Framework
The FedCM framework consists of three integrated components designed to identify and maintain optimal client groupings throughout the training process.
1. Personalized Gradient Path Similarity
Rather than comparing static model weights, FedCM focuses on the trajectory of gradients during a warm-up phase. By tracking the sequence of gradient updates ($g_i^t$) in the classification layer, the system captures how a model learns from a specific client's data. The similarity between two clients $i$ and $j$ is calculated using a cosine similarity matrix $S_{ij}$ based on these gradient paths. This approach provides a more distinct "fingerprint" of client data than final weight values alone.
2. Adaptive Graph-Based Clustering
To avoid the need for pre-specifying the number of clusters (a common requirement in K-Means), FedCM employs an adaptive graph-based approach. Clients are represented as nodes in a graph, with edges weighted by their gradient path similarity. A modularity-optimization algorithm is then used to detect natural clusters within this graph, allowing the system to scale and adapt to the underlying data structure automatically.
3. Update Angle-Based Migration
To handle distribution drift, FedCM monitors the "angle" of each client's updates relative to its cluster's average. If a client's update direction begins to deviate significantly, it is flagged as an outlier. The system then "migrates" this client to a more compatible cluster. This mechanism ensures that the model remains performant even as user behavior changes, without requiring access to raw client data.
Experimental Evaluation
FedCM was rigorously tested on four computer vision benchmarks: CIFAR-10, CIFAR-100, Fashion-MNIST (FMNIST), and SVHN. These datasets were partitioned to simulate extreme label skews (pathological and standard).
Key Results
- Accuracy Gains: On CIFAR-10 with pathological skew, FedCM achieved an accuracy of 90.94%, a 2% improvement over the state-of-the-art PACFL and a massive jump from the 38.30% achieved by standard FedAvg.
- Resilience to Drift: In scenarios where data drift was introduced mid-training, FedCM maintained near-peak performance. While other methods saw performance drops of up to 30‰, FedCM's degradation was limited to a mere 1‰, demonstrating its ability to "heal" the system through rapid client migration.
- Parameter-Free Efficiency: The adaptive clustering reached optimal performance without manual tuning of the cluster count, simplifying deployment in unknown environments.
Discussion and Insights
The success of FedCM highlights several critical insights for federated learning:
- Gradients as Trajectories: The path taken during training is more informative than the destination. Gradient paths create sharper cluster boundaries, making it easier to distinguish between clients with different data distributions.
- Classification Layer Focus: Focusing on the final classification layer is computationally efficient and acts as a strong proxy for task identity.
- Privacy-Preserving Drift Detection: Using update angles allows for robust drift detection with minimal overhead and no additional privacy risks, as it relies on information already shared during the standard FL update process.
Conclusion and Future Work
FedCM provides a robust solution for the twin challenges of heterogeneity and drift in federated learning. By leveraging gradient path similarity and dynamic migration, it ensures that clients are always grouped with peers who share similar data patterns. Future research will explore the application of FedCM to non-classification tasks, such as regression and generative modeling, and further optimize the communication overhead associated with tracking gradient paths during the warm-up phase.
Summary updated