In standard Federated Learning (FL), a central server coordinates a network of decentralized clients to train a shared global model. Algorithms like FedAvg operate on a clean mathematical assumption: each client's local data is independent and identically distributed (IID) drawn from the same global population.
In the real world, this assumption completely falls apart. Real-world users have highly idiosyncratic habits, leading to label distribution skew (or label heterogeneity). A specialized medical clinic might only see a handful of rare diseases, while a mobile phone user in a specific region might only type messages about local sports. As Sixu Hu et al. demonstrated in The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems, this mismatch in category diversity across nodes harms global model accuracy far more severely than simple variations in data volume.
When clients train on highly skewed local datasets, standard federated optimization algorithms diverge, causing the global model's performance to collapse. To understand why this happens—and how to fix it—we must look at how label skew distorts a neural network's internal representations.
The Mechanics of Failure: Local Priors and Feature Collapse
To understand why label skew breaks federated learning, consider a client training a local model on a dataset containing only images of cats, completely lacking any images of dogs.
When this client performs local gradient descent using a standard cross-entropy loss, two destructive phenomena occur simultaneously:
- Classifier Weight Drift: The final classification layer (the head) adapts to the local class prior. Because the client never sees a dog, the network learns that the probability of a dog appearing is zero. The weights corresponding to the "cat" class are amplified, while the weights for the missing "dog" class are suppressed or driven to random values.
- Feature Space Collapse: The feature extractor (the layers preceding the classifier) stops learning features that distinguish cats from dogs. It only learns features that distinguish different types of cats, or worse, it collapses all inputs into a narrow, poorly defined region of the latent space. As Xinting Liao et al. explain in HyperFed: Hyperbolic Prototypes Exploration with Consistent Aggregation for Non-IID Data in Federated Learning, this absence of samples causes the local feature spaces for missing classes to become highly distorted.
When the central server averages these locally trained models, it is not combining complementary insights. Instead, it is averaging models that have optimized completely incompatible decision boundaries. The resulting global model suffers from severe "model drift" and fails to generalize across the full global distribution. In extreme scenarios like one-shot federated learning, where clients communicate only once, Yiqun Diao et al. observe in Towards Addressing Label Skews in One-Shot Federated Learning that this distribution mismatch causes the aggregated model's performance to completely collapse.
Strategy 1: Decoupling Representation from Classification
Because the classifier head is highly sensitive to local class imbalances, a prominent line of research focuses on separating the feature extractor from the classification layer. The intuition is simple: keep the feature extractor collaborative, but treat the classifier with special care.
In Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Features, Xinyi Shang et al. exploit this insight by retraining the global classifier on the server side. Instead of relying on the biased local classifiers sent by clients, the server synthesizes privacy-preserving "federated features" to optimize a balanced global classifier head.
Taking decoupling a step further, Hong-You Chen et al. propose FED-ROD in On Bridging Generic and Personalized Federated Learning for Image Classification. They equip the network with two distinct prediction heads: a generic head optimized for global robustness and a personalized head tailored to the client's local distribution. This prevents the local class priors from polluting the global parameters.
Standard FL: [ Feature Extractor ] ---> [ Biased Local Classifier ] ---> Aggregated blindly
Decoupled FL: [ Feature Extractor ] ---> [ Global Head (Robust) ]
`--> [ Local Head (Personalized) ]
Architectural adjustments within the network can also inherently mitigate this drift. The authors of Understanding the Role of Layer Normalization in Label-Skewed Federated Learning discovered that applying feature normalization right before the classifier head acts as a critical buffer. By normalizing the latent representations, it prevents feature collapse and limits the ability of a skewed local dataset to over-amplify specific class weights.
Strategy 2: Global Anchors and Prototype Alignment
If a client lacks samples for certain categories, how can we prevent its feature space from collapsing? The answer lies in providing a "compass"—a global reference point that tells the local model where missing classes should reside in the latent space.
One way to achieve this is through global prototypes. In FedHAP: Federated Hashing with Global Prototypes for Cross-silo Retrieval, Meilin Yang et al. utilize aggregated, class-level hash codes as global anchors. These prototypes are distributed to clients, forcing local models to align their feature representations with a unified global structure, ensuring that missing classes retain their geometric meaning during local updates.
Similarly, Chunhui Du et al. introduce FedCMC in Contrast with Major Classifier Vectors for Federated Medical Relation Extraction with Heterogeneous Label Distribution. FedCMC extracts the most representative class vectors across different clients and uses them as anchors for contrastive learning. By penalizing local representations that stray too far from these global anchors, it prevents local models from overfitting to their skewed distributions.
Strategy 3: Statistical Correction and Data Reshaping
Instead of altering the model's architecture or latent space, we can address label skew by mathematically correcting the optimization objective or physically altering the data distribution.
Probability-Corrected Losses
When a client trains on a skewed distribution, the local loss function implicitly optimizes for local class frequencies rather than global ones. To counteract this, De-Chuan Zhan et al. introduce a statistically principled framework in Aligning model outputs for class imbalanced non-IID federated learning. They employ a probability-corrected loss that dynamically adjusts the local model's outputs based on the client's specific class priors, ensuring that identical inputs yield consistent predictions across different devices.
Proactive Data Reshaping
Alternatively, we can mitigate statistical skews before training even begins. Rather than accepting whatever data happens to reside on a client, systems can proactively balance the distribution.
- Client Selection: In Federated learning with class imbalance reduction, Miao Yang et al. use a gradient-based estimation trick to infer local label distributions without violating privacy. A Multi-Armed Bandit algorithm then selects a subset of participants whose combined data forms a balanced, IID-like distribution for that training round.
- Edge Offloading: In FlocOff: Data Heterogeneity Resilient Federated Learning with Communication-Efficient Edge Offloading, Mulei Ma et al. treat edge offloading as a data-reshaping tool. By strategically moving data samples from individual devices to nearby edge servers, they physically balance the local class distributions before the training process starts.
By combining architectural decoupling, prototype alignment, and statistical corrections, modern federated learning frameworks transform label distribution skew from a catastrophic bottleneck into a manageable engineering challenge.