Standard Federated Learning (FL) operates on a "one-size-fits-all" principle: every participating device must download, train, and upload the exact same global model. While this works in controlled environments, it creates a "rich-client" problem in the wild. As noted by Fatih İlhan and Gong Su in ScaleFL, standard protocols exclude resource-constrained hardware like low-power sensors or legacy mobile devices that cannot fit a massive foundation model into their local memory.
This exclusion leads to the "bucket effect," where the complexity of the global model is bottlenecked by the weakest participating device. To solve this, researchers are developing mechanisms that allow a single global model to be trained across a fleet of devices with vastly different RAM, CPU, and bandwidth capacities.
Sub-model Extraction and Ordered Dropout
The most direct way to accommodate a weak device is to give it a smaller piece of the model. Enmao Diao and Jie Ding pioneered this with HeteroFL, which adaptively distributes subnetworks of varying widths to different devices. Instead of training a 512-channel layer, a resource-constrained device might only train the first 128 channels.
To ensure these different-sized updates actually contribute to a coherent global model, Samuel Horváth and colleagues introduced FjORD. They use Ordered Dropout, which creates nested submodels within the main architecture. Because the dropout is "ordered," the most important features are packed into the smallest sub-networks, ensuring that even the updates from the weakest IoT sensors align with the core of the global model.
While width-based pruning is common, Kai Zhang argues in FEDEPTH that depth-wise decomposition is often more effective. By training sequential blocks of a model one by one, devices can contribute to deep architectures without ever needing to hold the entire stack in VRAM.
Knowledge Distillation as a Universal Translator
If devices are training fundamentally different architectures, simple weight averaging (FedAvg) fails. Researchers use Knowledge Distillation (KD) to bridge this gap. In FedGKT, Chaoyang He and Salman Avestimehr propose a bidirectional exchange: small edge models and a large server model "teach" each other using their respective outputs. This allows the server to aggregate "intelligence" rather than raw parameters.
Similarly, RaFL by Waqwoya Abebe and Ali Jannesari uses a tiny "knowledge network" that acts as a universal translator. Every device, regardless of its custom-tailored architecture, communicates through this shared network, allowing the federation to benefit from high-end hardware without leaving low-power nodes behind.
Heterogeneous Low-Rank Adaptation
For Large Language Models (LLMs), the focus has shifted to Parameter-Efficient Fine-Tuning (PEFT), specifically Low-Rank Adaptation (LoRA). However, even LoRA has a memory footprint that scales with the "rank" of the adapters.
FlexLoRA, proposed by Jiamu Bai, allows devices to choose a LoRA rank that fits their specific hardware. A workstation might use a rank of 64, while a phone uses a rank of 4. The server then synthesizes these varying updates into a full-size weight matrix. Raghav Singhal and Praneeth Vepakomma further refined this in FedEx-LoRA by introducing a residual error term that mathematically accounts for the discrepancy between local low-rank updates and the global model, achieving performance parity with centralized training.
Principal Components and Random Allocation
To avoid the "feature imbalance" that occurs when certain parts of a model are trained more frequently than others, researchers are exploring probabilistic methods. PriSM, developed by Yue Niu and Souvik Kundu, uses Singular Value Decomposition (SVD) to identify the "principal" sub-models. By sampling these components based on importance, they reduce local memory needs by 80% while maintaining near-full-model accuracy.
Alternatively, Shangchao Su and Bin Li propose FedRA, which randomly allocates different subsets of model layers to clients in each round. This ensures that every layer of a massive foundation model receives updates over time, even if no single client can ever host the entire model at once.
Balancing Personalization and Generalization
Hardware heterogeneity often correlates with data heterogeneity. A device's physical constraints (like a low-resolution camera) might influence the data it collects. PAGE by Qian Chen treats this as a "co-opetition" game, using reinforcement learning to balance a model's ability to generalize across the fleet with its ability to personalize for a specific device's hardware and data. Methods like kNN-Per by Othmane Marfoq simplify this by combining a global model with a local "memorization" layer, allowing even weak devices to achieve high local accuracy without the overhead of complex personalization algorithms.