Federated learning (FL) promises a decentralized future where machine learning models learn from private data distributed across millions of edge devices—like smartphones, medical sensors, and IoT hardware—without that data ever leaving the device.
In an ideal theoretical sandbox, every device participates equally. In the real world, however, a massive hardware disparity exists between a flagship smartphone and a budget device or an older sensor. When distributed optimization algorithms encounter this hardware heterogeneity, a systemic failure mode emerges: hardware-induced client exclusion bias.
When a system drops or marginalizes slow, low-memory, or poorly connected devices to keep training efficient, it doesn't just optimize performance. It alters the statistical representation of the global model, inadvertently creating an unfair, poorly generalizing system that favors affluent demographics with high-end hardware.
The Hidden Link Between Systems and Statistics
To understand why hardware-induced bias occurs, we must look at how systems heterogeneity couples with statistical heterogeneity. In distributed networks, a device's computational capacity is rarely independent of the data it holds. As Tian Li et al. and Anit Kumar Sahu et al. point out, hardware performance often correlates directly with specific user demographics, geographic locations, or socioeconomic factors.
Imagine training a predictive text or healthcare model across a global population. Users with older or less expensive devices may live in regions with limited infrastructure, speak minority dialects, or exhibit unique behavioral patterns. If the federated framework systematically excludes these slower devices, the final global model fails to learn from their data distributions.
[All Distributed Clients]
│
├── High-End Devices ──► Fast Updates ──► Strong Influence on Global Model
│
└── Low-End Devices ──► Timeouts/Drops ──► Zero/Weak Influence (Statistical Bias)
As Isidoros Tziotis et al. demonstrate, assuming computational speed is independent of local data is a critical flaw in standard FL theoretical models. When this assumption breaks down, system-level dropouts directly degrade the statistical representativeness of the trained model, resulting in what Adam Sadilek et al. warn can exacerbate real-world health and social inequities.
The Mechanisms of Resource-Based Exclusion
How exactly do low-resource devices get left behind? The exclusion manifests through three primary system bottlenecks:
1. The Straggler Problem and Rigid Timeouts
To prevent a few slow nodes (stragglers) from stalling an entire training round, central servers often implement strict scheduling or timeout windows. Devices that fail to complete their local training epochs within the window are dropped. Adnan Mahmood et al. and Behnaz Soltani et al. show that prioritizing high-speed devices to minimize training latency introduces severe selection bias. Even in asynchronous setups where nodes update at their own pace, Hossein Zakerinia et al. observe that high-performance nodes contribute far more frequent updates, drowning out the unique data characteristics of slower nodes.
2. The Memory Wall
As foundation models grow, they hit a hard physical constraint: on-device RAM. Yebo Wu et al. note that standard deep learning architectures often require more memory to train than budget smartphones or low-cost sensors possess. When a model is too large for a device's memory capacity, the device simply crashes or fails to initialize training, leading to complete demographic exclusion as detailed by Joerg Henkel et al..
3. Network and Energy Constraints
Real-world deployment constraints dictate that devices should only participate when they are idle, charging, and connected to unmetered, high-speed Wi-Fi. Lei Fu et al. emphasize that this creates a significant participant bias, ignoring users whose daily routines or living conditions do not align with these ideal hardware states. Furthermore, geographic disparities in network bandwidth mean users with limited upload speeds are effectively locked out of collaborative training, as highlighted by Han Liu et al..
The Consequence: Model Drift and Fairness Gaps
When low-resource clients are excluded or under-represented, the optimization objective shifts. Instead of minimizing the loss across the true population distribution, the global model minimizes the loss primarily for the subset of users with high-end hardware.
This causes model drift. The global model's parameters track the data distributions of dominant, high-performance devices. Data collected by Daoyuan Chen et al. in real-world cross-device deployments confirms that this imbalance creates a severe hardware-driven fairness gap, resulting in a significant drop in accuracy for the least-advantaged clients.
Compounding this problem, Gyudong Kim et al. show that models can overfit to the physical characteristics of prevalent hardware itself (such as specific camera sensors or image processors), making the global model fragile and poorly generalized when deployed on marginalized device configurations.
This creates a fundamental ethical and functional dilemma. As Feilong Zhang et al. note in the context of collaborative healthcare, a local facility's IT budget or an individual's device tier shouldn't dictate the quality of the AI-driven insights they receive.
Designing for Inclusivity
To build truly responsible, representative federated systems, researchers are developing mechanisms that decouple a device's physical constraints from its ability to contribute to the global model. Rather than forcing a "one-size-fits-all" architecture or dropping slow nodes, modern frameworks adapt the workload to the client's capability.
Ordered Dropout and Sub-Model Training
Instead of training the full global model on every device, the server can distribute smaller, nested sub-models tailored to each client's memory and compute tier. As argued by Samuel Horváth, techniques like Ordered Dropout allow low-end IoT devices and flagship smartphones to train the same global model simultaneously. The authors of FedRolex expand on this by using rolling sub-model extraction to ensure that all parameters of the global model are evenly trained across diverse hardware tiers, preventing parameter stagnation.
Global Model Layers: [ L1 ][ L2 ][ L3 ][ L4 ][ L5 ]
▲ ▲ ▲ ▲ ▲
High-End Client: [ L1 ][ L2 ][ L3 ][ L4 ][ L5 ] (Trains full model)
Low-End Client: [ L1 ][ L2 ] (Trains sub-model via dropout)
Heterogeneous Fine-Tuning and Optimization
For large foundation models, frameworks like those proposed by Matt Barnes et al. allow devices to train using varying Low-Rank Adaptation (LoRA) ranks based on their hardware limits, aggregating these heterogeneous updates seamlessly at the server. To lower the computational entry barrier even further, Gwen Legate demonstrates that "warming up" a model with high-resource clients before transitioning to memory-efficient zeroth-order optimization allows previously excluded low-resource edge devices to safely participate.
Architecture-Agnostic Collaboration
To completely break the architectural constraints of standard FL, Disha Makhija introduces frameworks that leverage knowledge distillation. By aligning instance-level representations using a small, independent dataset, clients with entirely different hardware capabilities and neural network architectures can collaborate without sharing weights, ensuring that data diversity is preserved without penalizing resource-constrained participants.
By shifting the focus from pure training speed to capability-aware participation, these methods ensure that federated learning can live up to its promise: creating robust, fair, and globally representative models that leave no client behind.
Key Foundational Work
Towards Flexible Device Participation in Federated Learning for Non-IID Data (2020) — contributed 2 concept elements to this direction.
On the Convergence of Federated Optimization in Heterogeneous Networks (2018) — contributed 2 concept elements to this direction.
Expanding the Reach of Federated Learning by Reducing Client Resource Requirements (2018) — contributed 1 concept elements to this direction.