In standard Federated Learning (FL), the goal is to train a global model by averaging the weights of local models trained on decentralized devices. However, when data is "non-IID"—meaning every device sees a different, biased slice of the world—this weight-averaging approach breaks down. Local models "drift" toward their specific data skews, and averaging their divergent weights often results in a global model that performs poorly for everyone.
A growing body of research suggests that the solution lies not in forcing models to have similar parameters, but in forcing them to produce similar representations. By aligning the internal feature spaces of local models with a global reference, researchers are building federated systems that are robust to data heterogeneity, label noise, and catastrophic forgetting.
The Problem: Weight Divergence vs. Semantic Drift
When a local model trains on a biased dataset (e.g., a hospital that only sees one type of pathology), its weights move in a direction that minimizes local loss but destroys global utility. Traditional methods like FedProx try to fix this by penalizing the Euclidean distance between local and global weights.
However, Qinbin Li, Bingsheng He, and Dawn Song argue in Model-Contrastive Federated Learning that weight-level similarity is a poor proxy for model behavior. Two neural networks can have very different weights but produce nearly identical feature embeddings. Their framework, MOON, shifts the focus to the latent representation space, using contrastive learning to ensure that a local model’s features are closer to the global model’s features than to its own previous (and potentially drifted) iterations.
Mechanism 1: Contrastive Anchoring
The most prominent mechanism for this alignment is contrastive loss. In this setup, the global model acts as a "teacher" or an anchor. During local training, the client doesn't just learn to classify its data; it also learns to make its feature embeddings match those of the global model.
Hesham Mostafa demonstrated in Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameters that forcing this alignment prevents local models from specializing too heavily on client-specific noise. Similarly, Yingguang Yang and colleagues developed FedACK, which uses contrastive loss to align feature spaces even across different model architectures and languages, allowing the global model to aggregate knowledge from highly fragmented sources like social bot data FedACK: Federated Adversarial Contrastive Knowledge Distillation for Cross-Lingual and Cross-Model Social Bot Detection.
Mechanism 2: Global Class Prototypes
While MOON and FedACK align representations instance-by-instance, another powerful approach uses prototypes—the average feature vector for a specific class across the entire network.
In Boosting Federated Learning Convergence with Prototype Regularization, Yu Qiao 0004 and Huy Q. Le introduce FedPR. Instead of downloading a full model, clients can download "global prototypes" for each class. Local training is then regularized to minimize the distance between local features and these global anchors. This ensures that even if a client only sees a few examples of a "dog," its internal definition of a "dog" remains consistent with the global consensus.
This concept is extended in several ways:
- Multi-Prototypes: Yu Qiao 0004 and Apurba Adhikary propose MP-FedCL, which uses multiple prototypes per class to capture diverse feature distributions within a single category MP-FedCL: Multi-Prototype Federated Contrastive Learning for Edge Intelligence.
- Dynamic Graphs: Zaipeng Xie applies prototype regularization to evolving graphs in FedDGL, preventing "catastrophic forgetting" as the graph structure changes over time FedDGL: Federated Dynamic Graph Learning for Temporal Evolution and Data Heterogeneity.
- Feature Norms: Ke Hu and Weidong Qiu found that even simply aligning the magnitude (norm) of class-specific features can mitigate drift, leading to massive accuracy gains in FNR-FL Feature Norm Regularized Federated Learning: Utilizing Data Disparities for Model Performance Gains.
Robustness to Noise and Label Errors
Representation alignment doesn't just solve the non-IID problem; it also acts as a shield against "Federated Label-Noise." Local clients often have messy or incorrect labels. If a local model overfits to these errors, it creates "poisoned" updates.
Yuxin Tian and Tongliang Liu observe that global models, being aggregates of many clients, memorize noise much slower than individual local models. In Learning Locally, Revising Globally: Global Reviser for Federated Learning with Noisy Labels, they use the global model's feature representations to "sniff out" local label errors. By penalizing deviations from the global representation, the local model is forced to prioritize generalizable features over memorizing client-specific noise.
Why It Works: The Shared Semantic Space
The core intuition behind this entire research direction is the creation of a shared semantic space. In standard FL, we hope that averaging weights will implicitly create this space. In representation-aligned FL, we explicitly build it.
By using contrastive losses (Chaoning Zhang, Logit Calibration and Feature Contrast for Robust Federated Learning on Non-IID Data) or intermediate layer regularization (Ye Lin Tun, Federated Learning with Intermediate Representation Regularization), these methods ensure that the "meaning" of data remains consistent across the network. Whether a model is looking at a medical image in a rural clinic or a high-tech hospital (Jeffry Wicaksana, FCA: Taming Long-tailed Federated Medical Image Classification by Classifier Anchoring), the underlying feature extractor is anchored to a global definition of the task, making the resulting updates far more compatible for aggregation.