Abstract
Federated learning (FL) enables collaboratively training a model while keeping the training data decentralized and private. However, one significant impediment to training a model using FL, especially large models, is the resource constraints of devices with heterogeneous computation and communication capacities as well as varying task sizes. Such heterogeneity would render significant variations in the training time of clients, resulting in a longer overall training time as well as a waste of resources in faster clients. To tackle these heterogeneity issues, we propose the Dynamic Tiering-based Federated Learning (DTFL) system where slower clients dynamically offload part of the model to the server to alleviate resource constraints and speed up training. By leveraging the concept of Split Learning, DTFL offloads different portions of the global model to clients in different tiers and enables each client to update the models in parallel via local-loss-based training. This helps reduce the computation and communication demand on resource-constrained devices and thus mitigates the straggler problem. DTFL introduces a dynamic tier scheduler that uses tier profiling to estimate the expected training time of each client, based on their historical training time, communication speed, and dataset size. The dynamic tier scheduler assigns clients to suitable tiers to minimize the overall training time in each round. We first theoretically prove the convergence properties of DTFL. We then train large models (ResNet-56 and ResNet-110) on popular image datasets (CIFAR-10, CIFAR-100, CINIC-10, and HAM10000) under both IID and non-IID systems. Extensive experimental results show that compared with state-of-the-art FL methods, DTFL can significantly reduce the training time while maintaining model accuracy.
Introduction
Federated Learning (FL) typically requires all participating devices (clients) to train the same global model locally. In real-world deployments, these devices are highly heterogeneous, ranging from high-end edge servers to low-power IoT sensors. This disparity creates the "straggler problem," where the entire training round is bottlenecked by the slowest device. Furthermore, as deep learning models grow larger (e.g., ResNets), many resource-constrained devices lack the memory or compute power to participate. Existing solutions like Split Learning (SL) help by splitting the model between client and server, but they often introduce sequential bottlenecks or use static splits that don't adapt to a device's changing resources.
The DTFL Framework
The Dynamic Tiering-based Federated Learning (DTFL) framework addresses these hurdles through three core components:
- Tiered Split Learning: Instead of a single split point, DTFL defines $M$ tiers. Clients in lower tiers (the "stragglers") keep only the first few layers of the model, offloading the rest to the server. Higher-tier clients keep more of the model locally.
- Local-Loss-Based Training: To avoid the sequential "wait-and-see" bottleneck of standard Split Learning, DTFL uses auxiliary networks (small fully connected layers) at the client side. This allows clients to compute a local loss and update their portion of the model in parallel with the server's updates.
- Dynamic Tier Scheduler: This component profiles each client’s communication speed, computation capacity, and dataset size. Using an Exponential Moving Average (EMA) of historical performance, it dynamically reassigns clients to the optimal tier each round to minimize the overall training time ($T_{max}$).
By using auxiliary networks for local loss, DTFL breaks the "lock-step" nature of split learning. The server and clients can work simultaneously, which is a major departure from standard Split Learning (SL) or Split Federated Learning (SFL).
Experimental Evaluation
The framework was evaluated on four widely used image classification benchmarks: CIFAR-10, CIFAR-100, CINIC-10, and HAM10000 (a medical dataset of skin lesions). These were tested under both IID and non-IID settings to simulate realistic data distributions.
Key Results
DTFL consistently outperformed state-of-the-art methods like FedYogi, SplitFed (SFL), and FedGKT across all tested datasets and model architectures (ResNet-56 and ResNet-110).
- Massive Speedups: Compared to standard FedAvg, DTFL reduced the training time to reach 80% accuracy on CIFAR-10 (using ResNet-110) by approximately 80%.
- Scalability: In tests with up to 200 clients, DTFL maintained its performance advantage, showing that the scheduling overhead remains low even as the network grows.
- Robustness: The system successfully adapted to dynamic environments where client resources (CPU/bandwidth) changed every 50 rounds.
Discussion and Conclusion
DTFL navigates the non-linear trade-off between computation and communication in real-time. While offloading more layers reduces a client's computation time, it can increase communication overhead due to the size of intermediate feature maps. The dynamic scheduler is essential for navigating this trade-off.
Furthermore, DTFL is compatible with privacy-preserving techniques like Distance Correlation and Patch Shuffling, which decorrelate data from activations with minimal impact on accuracy. Future work may explore more sophisticated auxiliary architectures to further improve the gradient signal for client-side models and address communication bottlenecks in extremely bandwidth-starved environments.
Summary updated