Abstract
Federated learning (FL) collaboratively trains models across multiple clients without direct data access. However, data heterogeneity often forces a trade-off between generic performance (a global model for all) and personalized performance (specific models for each client). We propose Federated Robust Decoupling (Fed-RoD), a framework that bridges this gap by explicitly decoupling a model's dual duties. By using a shared feature extractor with two specialized prediction heads—one for generic tasks using balanced risk minimization and one for personalized tasks using empirical risk—Fed-RoD achieves state-of-the-art results in both domains simultaneously.
The Dilemma of Data Heterogeneity
In Federated Learning, clients often possess non-IID (non-identically and independently distributed) data. This heterogeneity creates a fundamental tension: should the server optimize for a single global model that generalizes well to new users (Generic FL), or should it prioritize models tailored to each client's specific data distribution (Personalized FL)?
Historically, these have been treated as separate objectives. Generic FL algorithms like FedAvg often struggle with local client performance, while personalized approaches may fail to produce a robust global model. Interestingly, the authors observe that standard generic algorithms actually generate strong personalized models during local training, but these are typically discarded in favor of the global average at the end of each round.
Federated Robust Decoupling (Fed-RoD)
To resolve this conflict, Fed-RoD introduces a "two-loss, two-predictor" framework. The architecture consists of a shared feature extractor and two distinct prediction heads:
- Generic Head ($h^G$): This head is trained using Balanced Risk Minimization (BRM). By employing a Balanced Softmax loss, it learns a representation that treats all classes fairly, regardless of local label skew. This head is aggregated at the server to form the global model.
- Personalized Head ($h^P$): This lightweight module is trained to minimize the client's specific Empirical Risk. It adapts the generic representation to the local data distribution.
The framework offers two implementation paths for the personalized predictor:
- Fed-RoD (linear): Uses a simple local linear layer for personalization.
- Fed-RoD (hyper): Utilizes a hypernetwork that generates personalized weights based on the client's class distribution. This enables "zero-shot" personalization for new clients who were not part of the training set.
Key Insights and Results
- Implicit Regularization: The process of re-initializing local models with the global average acts as a powerful regularizer, preventing overfitting to small local datasets more effectively than standard $L_2$ penalties.
- Decoupling Benefits: By separating the "generic" duty (balanced representation) from the "personalized" duty (local bias adaptation), Fed-RoD prevents objective competition, allowing both to reach peak performance.
- State-of-the-Art Performance: Evaluated on CIFAR-10, CIFAR-100, and EMNIST, Fed-RoD consistently outperformed both generic (FedAvg, FedProx) and personalized (Ditto, pFedMe) baselines. On CIFAR-10 with high skew, Fed-RoD achieved 76.9% generic accuracy (vs. FedAvg's 68.6%) and reached personalized scores as high as 94%.
Conclusion
Fed-RoD demonstrates that generic and personalized performance are not mutually exclusive. By decoupling these tasks into specialized heads on a shared backbone, federated systems can provide both a robust global model for new users and highly accurate personalized models for existing ones, effectively bridging the two major tracks of federated learning research.
Summary updated