In federated learning, the standard recipe is simple: distributed clients train local models on their private data, and a central server averages their weights via algorithms like FedAvg. However, this approach breaks down when data is non-independent and identically distributed (non-IID) across devices. When clients have highly skewed label distributions, their local optimization trajectories diverge. This phenomenon, known as client drift, means that averaging the weights of these divergent models results in a global model that performs poorly across all domains.
Why does this happen? Without a shared reference point, each client's feature extractor distorts the latent space to fit its local data distribution. A feature vector that represents a "cat" on Client A might land in the exact same region of the embedding space where Client B maps a "dog". Averaging their weights creates a semantic mismatch.
To solve this, a powerful paradigm has emerged: instead of forcing alignment solely through model parameters, we can align models in the representation space using class prototypes.
Prototypes as Shared Semantic Anchors
Imagine a group of cartographers trying to map the world, but each is stuck in a different country with a different compass calibration and coordinate system. If they just average their map coordinates, the resulting map is a blurry mess. Prototype alignment gives everyone a set of shared landmarks (e.g., the Eiffel Tower, the Pyramids). Even if their local maps are incomplete, they can align their local coordinate systems to these landmarks.
In machine learning, a prototype is simply the mean feature vector computed for a specific category. In a prototype-based federated framework, such as the one proposed by Guodong Long et al. in Federated Prototype Learning across Heterogeneous Clients (FedProto), clients compute local prototypes for each class they possess. The server aggregates these local vectors to form global prototypes, which are then distributed back to the clients.
Client A (Data: Cats) ───> Local Cat Prototype ───┐
▼
Client B (Data: Dogs) ───> Local Dog Prototype ───┼─> Server Aggregation ─> Global Prototypes
▲ (Shared Anchors)
Client C (Data: Birds) ───> Local Bird Prototype ───┘
These global prototypes act as a universal coordinate system. During local training, clients minimize a regularization loss that penalizes the distance between their local feature representations and the corresponding global prototypes. As Jian Xu et al. demonstrate in Personalized Federated Learning with Feature Alignment and Classifier Collaboration, this geometric anchoring allows individual nodes to perform more local training iterations without drifting away from the shared global structure, significantly accelerating communication efficiency.
Geometric Regularization and Contrastive Alignment
How exactly do these prototypes prevent representation collapse? When a client lacks samples for certain classes, its local model can develop extreme biases, misclassifying rare categories into dominant local classes. Global prototypes provide a structural anchor that prevents local models from developing these extreme biases, ensuring that locally generated representations remain compatible with categories they have not personally encountered. This mechanism is highlighted by Jiale Liu et al. in Prototype-Based Layered Federated Cross-Modal Hashing.
To maximize the discriminative power of the shared latent space, many frameworks employ contrastive learning. For instance, Yue Tan et al. in Federated Learning from Pre-Trained Models: A Contrastive Learning Approach utilize global and local class prototypes within a contrastive framework. Features are pulled toward their true global class prototype and pushed away from imposter prototypes:
$$\mathcal{L}_{\text{contrastive}} = -\log \frac{\exp(\text{sim}(f(x), P_c) / \tau)}{\sum_{j} \exp(\text{sim}(f(x), P_j) / \tau)}$$
where $f(x)$ is the local feature extraction, $P_c$ is the target global prototype, and $P_j$ represents other class prototypes. This contrastive alignment maintains stability and improves accuracy even as the number of clients scales up.
In unsupervised or clustering contexts, like the graph-level clustering network proposed by Jingxin Liu 0005 et al. in Federated Graph-Level Clustering Network, this alignment is achieved by minimizing the Kullback-Leibler (KL) divergence between local cluster distributions and global consensus prototypes. This feedback loop ensures that local models align with global structural patterns without sacrificing local data autonomy.
Fixed Semantic Anchors via Multimodal Models
While aggregating empirical prototypes from clients is highly effective, it still introduces a dependency on client data distributions and requires communication overhead. A powerful evolution of this idea bypasses prototype aggregation entirely by leveraging pre-trained multimodal models like CLIP.
Instead of calculating empirical averages from client features, the server can generate fixed textual embeddings for all possible classes using a pre-trained language model. As Yunlu Yan et al. show in On the Importance of Language-driven Representation Learning for Heterogeneous Federated Learning, these language-driven embeddings serve as immutable global targets, ensuring that all participating clients optimize their local encoders toward a shared, semantically consistent representation regardless of their specific local data.
Similarly, Lei Meng et al. in Cross-Silo Feature Space Alignment for Federated Learning on Clients with Imbalanced Data use these fixed text embeddings as universal anchors to regularize local feature representations. Because the anchors are fixed and semantically rich from the start, clients can map local features to a globally consistent space even if they suffer from extreme class imbalances or completely lack certain categories.
Breaking the Architecture Barrier
Perhaps the most profound advantage of prototype alignment is its independence from model architecture. Traditional federated learning requires every client to run the exact same neural network because you cannot average the weights of a ResNet with those of a Vision Transformer.
Because prototypes are simply vectors residing in the final embedding space, prototype alignment decouples collaborative learning from the underlying model parameters. As long as different local models project their final representations into a shared dimensionality, they can communicate and align via global prototypes.
This property allows high-power servers and low-power edge devices to collaborate seamlessly using entirely different neural network structures. By shifting the focus from weight matching to representation-centric collaboration, prototype alignment provides a robust, scalable solution to the dual challenges of data and hardware heterogeneity.