Heterogeneity Aware Model Partitioning for Distributed Learning

46 concepts44 papers

Distributed training and inference algorithms traditionally rely on a convenient fiction: the homogeneity assumption. Whether deploying data-parallel training across a data center cluster or orchestrating federated learning across thousands of edge devices, standard frameworks assume that every participating node can comfortably store and compute the exact same model architecture.

In the real world, hardware is a messy, mismatched collection of generations. A single network might contain flagship AI accelerators, older "whimpy" GPUs, and low-power IoT sensors. If you force a large model onto this heterogeneous fleet using standard methods, you encounter a severe dilemma. You must either restrict the global model's size to fit the weakest participant—wasting the potential of high-end hardware—or exclude low-resource devices entirely, sacrificing valuable data diversity. In federated learning, this is known as the "rich-client" problem, as highlighted by Fatih İlhan et al. in ScaleFL: Resource-Adaptive Federated Learning with Heterogeneous Clients. Furthermore, as Ruixuan Liu argues in No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices, excluding resource-constrained devices introduces severe bias and degrades fairness.

Heterogeneity-Aware Model Partitioning resolves this dilemma. Instead of treating a neural network as an indivisible monolithic block, this paradigm slices, prunes, or distributes sub-components of the model dynamically. By tailoring the computational workload to the precise memory, compute, and network profile of each device, it allows highly mismatched hardware to collaboratively train or execute state-of-the-art models.


Slicing by Width: Nested Subnetworks and Dropout

One intuitive way to scale a model down for a weaker device is to make it "narrower" by reducing the number of active channels or hidden dimensions.

Global Model Width:  [====================================] (High-End Device)
Subnetwork Tier 2:   [========================]            (Mid-Range Device)
Subnetwork Tier 1:   [============]                        (Low-End Device)

Enmao Diao et al. formalize this in HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients. Instead of distributing the full model, the central server decomposes it into a strict hierarchy of subnetworks with varying channel widths. A low-end device receives and trains only the leftmost slice of the weight matrices. Crucially, these subnetworks overlap: the parameters trained by the smallest device are the exact same parameters that form the "core" of the largest model. When the server aggregates updates, the core parameters benefit from the data of all devices, while the wider, specialized parameters are optimized exclusively by high-performance nodes.

To make this width reduction seamless, the authors of FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout introduce Ordered Dropout. Instead of dropping neurons completely at random (as in standard dropout), Ordered Dropout enforces a strict right-to-left pruning order. This ensures that the submodels are neatly nested inside one another, preserving structural consistency across the fleet.

Rather than using static width tiers, recent methods determine neuron importance dynamically. For instance, Ji Liu et al. develop Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout, which ranks individual neurons using importance metrics derived from Hessian matrices. Devices with lower computational capacity are assigned customized, highly optimized sub-networks, ensuring that communication and compute budgets are balanced without sacrificing accuracy.


Slicing by Depth: Early Exits and Layer Freezing

An alternative to narrowing a model is making it "shallower." Because memory consumption during training scales heavily with the number of layers (due to storing intermediate activations), reducing model depth is an exceptionally effective way to accommodate memory-constrained hardware.

One approach embeds auxiliary prediction heads directly into the network architecture. Ji Wang et al. leverage this in Semi-HFL: semi-supervised federated learning for heterogeneous devices by constructing a multi-branch model with early-exit layers. Low-resource clients only download and train the shallow, early layers of the model up to a specific early exit. More powerful clients train the deeper branches. Minhui Xue et al. outline a similar strategy in RAI4IoE: Responsible AI for Enabling the Internet of Energy, using early-exit layers paired with self-distillation to ensure that even partial network execution yields accurate local outputs.

Input -> [ Layer 1 ] -> [ Layer 2 ] -> Early Exit Head (Weak Device stops here)
            ↓              ↓
         [ Layer 3 ] -> [ Layer 4 ] -> Final Output Head (Powerful Device completes)

Alternatively, depth-based partitioning can be managed over time through sliding windows. Joerg Henkel et al. introduce Successive Layer Training (SLT) in Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devices. Instead of forcing a device to process the entire depth of a network simultaneously, SLT uses adjustable layer-freezing windows. A memory-constrained device freezes the majority of the network and only computes gradients for a small, active window of layers. Over successive rounds, this window shifts through the network, allowing the device to eventually contribute to the optimization of the entire model without ever exceeding its local memory ceiling.


Heterogeneous Pipeline and Split Parallelism

When a model is too massive to fit on any single device—such as a modern Large Language Model (LLM)—the network must be split across multiple nodes. However, traditional pipeline parallelism divides layers evenly. If Node A is twice as fast as Node B, Node A will spend half its time idling, waiting for Node B to finish its forward or backward pass.

To eliminate these bottlenecks, heterogeneity-aware pipeline frameworks size each stage proportionally to the node's hardware capacity. Jay H. Park et al. demonstrate this optimization in HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism. By combining pipelined model parallelism with data parallelism, their framework partitions layers across heterogeneous devices by balancing computational load against communication costs. Similarly, the authors of HPipe: Large Language Model Pipeline Parallelism for Long Context on Heterogeneous Cost-effective Devices utilize dynamic programming to assign model layers based on a cluster's specific computational capacities and interconnect bandwidths, preventing faster devices from idling.

Uniform Split (Bottleneck):
GPU 1 (Fast): [ Layer 1-4 ] ---> Idle Waiting...
GPU 2 (Slow): [ Layer 5-8 ] ------------=========> Processing...

Heterogeneous Split (Balanced):
GPU 1 (Fast): [ Layer 1-6 ] --------=> Processing...
GPU 2 (Slow): [ Layer 7-8 ] --------=> Processing...

In edge-computing scenarios, this concept manifests as Split Learning, where a model is cut between a weak client device and a powerful cloud server. Mingjin Zhang et al. in Resource-efficient Parallel Split Learning in Heterogeneous Edge Computing and Guangyu Zhu et al. in ESFL: Efficient Split Federated Learning over Resource-Constrained Heterogeneous Wireless Devices show that the "cut layer" should not be static. By dynamically shifting the split point based on the client's instantaneous compute capacity and wireless network bandwidth, the system ensures that the client only processes a workload it can handle, pushing the remaining heavy lifting to the server. Yiannis Papageorgiou notes in Collaborative Split Federated Learning with Parallel Training and Aggregation that moving away from uniform layer boundaries toward per-device splitting strategies is essential for maximizing throughput in highly diverse edge networks.


Fine-Grained and Parameter-Efficient Adaptation

While splitting a model at coarse layer boundaries is effective, it can still fail to fully utilize fragmented memory and compute resources. Recent research has pushed partitioning down to sub-layer granularities and parameter-efficient modules.

For example, Dimitrios Kafetzis et al. address this in Large Language Model Partitioning for Low-Latency Inference at the Edge by breaking down Transformer layers at the attention head level. Instead of migrating entire layers, their heuristic resource allocation algorithms dynamically migrate individual attention heads between edge devices during inference, unlocking near-optimal execution speeds across fragmented hardware.

Coarse Layer Split:       [   Transformer Layer 1   ] -> [   Transformer Layer 2   ]
                                    ↓                                    ↓
Fine-Grained Head Split:  [Head 1][Head 2][Head 3]       [Head 4][Head 5][Head 6]
                            (Device A)     (Device B)      (Device A)     (Device C)

Another frontier applies heterogeneity-aware partitioning to Low-Rank Adaptation (LoRA). Rather than fine-tuning full parameter matrices, frameworks like Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation, developed by Ping Liu et al., allocate specific LoRA layers to clients based on available memory and layer importance. Non-assigned layers are frozen, drastically reducing the memory overhead of intermediate activations and optimizer states. Similarly, Arian Raje proposes Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuning, which decomposes weight updates into multiple independent low-rank heads. Resource-constrained devices train only a subset of these heads, maintaining global model consistency while minimizing the local compute footprint.


Algorithmic Orchestration and Optimization

Determining the optimal subnetwork configuration or pipeline split point across hundreds of mismatched devices is a complex optimization problem. If the server spends too much time calculating splits, or if the communication overhead of sending custom architectures outweighs the compute savings, the system stalls.

To manage this orchestration efficiently, systems employ automated planners and clustering techniques:

By treating hardware heterogeneity as a primary design constraint rather than an obstacle, heterogeneity-aware model partitioning shifts the paradigm of distributed deep learning. Slicing models along width, depth, or parameter dimensions ensures that no device is left behind, unlocking the collective power of diverse hardware fleets.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_heterogeneous_model_partitioning_ca79a692,
  title = {heterogeneous model partitioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-model-partitioning/rdr_ca79a692459948da859914870c4082bf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_heterogeneous_model_partitioning_ca79a692_v1,
  title = {heterogeneous model partitioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-model-partitioning/rdr_ca79a692459948da859914870c4082bf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake