Standard Federated Learning (FL) relies on a fundamental assumption: every participant must use the exact same neural network architecture. Algorithms like Federated Averaging (FedAvg) work by element-wise averaging of model weights. However, this "architectural homogeneity constraint" creates a significant bottleneck in real-world deployments. A high-end server and a low-power IoT sensor cannot easily share weights if their models have different layers, widths, or depths.
To overcome this, researchers have developed methods that move away from synchronizing parameters and toward synchronizing knowledge.
The Weight Averaging Wall
In a typical FL setup, the server sends a global model to clients, who train it on local data and send back updated weights. As noted by Lin et al. in Ensemble Distillation for Robust Model Fusion in Federated Learning, this fails when hardware capabilities vary. A smartphone might handle a ResNet-50, while a simple sensor might only support a tiny three-layer MLP. Because their weight matrices have different dimensions, they cannot be averaged.
Even when architectures match, Wu et al. argue in Heterogeneous Model Reuse via Optimizing Multiparty Multiclass Margin that forcing organizations to adopt a uniform architecture ignores the reality of specialized legacy systems. The challenge is to let clients "be themselves" while still learning from the collective.
Distillation as a Lingua Franca
The most prominent solution to architectural heterogeneity is Knowledge Distillation (KD). Instead of sharing weights, clients share their models' "opinions" on a common dataset.
If two models—one a Transformer and one a CNN—both look at a picture of a cat, they will produce different internal weights, but their output probability vectors (e.g., 90% cat, 5% dog) should be similar. By using these outputs as targets, a student model can learn the behavior of a teacher model without needing to match its structure.
Huang et al. propose a "dual-teacher" approach in FCCL, where models learn from the group's collective knowledge on public data while retaining local expertise. Similarly, Lin et al. use ensemble distillation on the server to fuse predictions from diverse client models, reaching target accuracies faster than traditional weight-based methods.
Elastic and Nested Architectures
Another approach is to design a single global model that is "elastic." Instead of different architectures, clients train different subsets of the same large model.
In HeteroFL, Diao et al. introduce a framework that distributes subnetworks of varying widths to clients based on their computational power. A powerful device trains the full width, while a weaker device trains a slimmed-down version. The server then aggregates these overlapping slices.
Horváth et al. extend this concept in FjORD using Ordered Dropout. By training nested submodels, they ensure that the most important features are packed into the smallest sub-architectures, allowing even the most resource-constrained IoT devices to contribute to the core of the global model. For Large Language Models, Bai proposes FlexLoRA, which allows devices to use different Low-Rank Adaptation (LoRA) sizes, synthesizing them into a full-size weight update at the server.
Communicating Through Prototypes
When models are too different even for subnetwork extraction, researchers look for shared representations. In FedKTL, Zhang uses a server-side pre-trained generator to act as a "knowledge bridge." Clients share compact class prototypes—mean feature representations of their data—rather than weights. This allows the server to transfer knowledge back to clients regardless of their local architecture.
This shift is also visible in specialized domains like Knowledge Graphs. Zhang et al. show in FEDR that aggregating relation embeddings instead of entity embeddings allows for collaboration even when clients use different graph neural network variants or scoring functions.
By moving the point of collaboration from the implementation (weights) to the information (outputs and prototypes), these frameworks enable a more inclusive and flexible federated ecosystem.