Speed Up Federated Learning in Heterogeneous Environment: A Dynamic Tiering Approach

Seyed Mahmoud Sajjadi Mohammadabadi, Syed Zawad, Feng Yan, Lei YangSubmission to ICLR 2024 · Withdrawn

TL;DR

Training large deep networks across heterogeneous edge devices usually grinds to a halt because slow "straggler" clients delay every round. Dynamic Tiering-based Federated Learning (DTFL) solves this by dynamically partitioning the model into tiers: slower devices train only a tiny initial portion of the network and offload the rest to the server, while local auxiliary loss heads allow clients and the server to train simultaneously in parallel.

What Problem Does This Paper Address?

Federated learning (FL) enables distributed clients to train a shared model collaboratively without centralizing raw data. However, deploying large models (like ResNet-56/110 or Transformers) onto mobile and IoT devices faces two major hurdles:

  1. Device Heterogeneity: Clients differ widely in computation power, network bandwidth, and dataset sizes. The slowest client (straggler) dictates round duration, leaving faster devices idling.
  2. Resource Constraints: Edge devices simply lack the memory and compute capacity to train deep models locally.

While standard Split Learning (SL) offloads layers to the server, it typically forces sequential execution (clients wait for backpropagated gradients from the server) and uses static split points that ignore changing network and hardware conditions.

Methods

DTFL blends split learning, tier-based scheduling, and decoupled greedy training:

  • Decoupled Local-Loss Training: Each tier splits the global model into a client-side model $w_c^m$ and a server-side model $w_s^m$. Clients attach a lightweight auxiliary head (average pooling + fully connected layer) to their intermediate representations to compute local error signals. This eliminates the need to wait for backpropagation from the server, allowing clients and the server to update weights in parallel.
  • Tier Profiling: Model execution time varies consistently across tiers regardless of hardware capacity. DTFL exploits this by profiling normalized relative training times across model tiers just once. Combined with exponential moving averages (EMA) of historical round times, network bandwidth, and dataset batch sizes, the server can accurately predict any client's runtime in any tier.
  • Dynamic Tier Scheduler: In each round, the scheduler estimates client completion times and identifies the minimum possible straggler latency ($T_{\max}$). It assigns each client to the deepest tier (maximizing client compute and minimizing server offload) whose predicted completion time stays below $T_{\max}$.
  • Model Aggregation: After each round, client and server sub-models are stitched and averaged into the unified global model, which is then sliced and redistributed according to the next round's tier assignments.
Client (Tier m)                      Server
[Input Data] ──> [Client Layers] ──> [Intermediate z_i] ──> [Server Layers] ──> Global Loss
                        │                                          │
                 [Auxiliary Head]                                  ▼
                        │                                    Server Update
                        ▼
                  Local Update
             (Runs in Parallel!)

Datasets

The authors evaluated DTFL on four image classification benchmarks under both I.I.D. and non-I.I.D. (Dirichlet distribution with concentration parameter $\alpha = 0.5$) settings:

  • CIFAR-10 & CIFAR-100
  • CINIC-10
  • HAM10000 (dermatoscopic skin lesion dataset)

The authors explicitly avoided LEAF benchmarks because their tasks are too small to properly benchmark large CNN architectures like ResNet-56 and ResNet-110. No new datasets were introduced.

Key Insights and Ideas

  • Decoupling Communication from Backprop: Standard split learning suffers from high latency because forward and backward passes alternate synchronously over the network. Local auxiliary classifiers allow forward activations to travel to the server while the client immediately backpropagates locally.
  • Offloading is Not Monotonic: Pushing more layers to the server reduces client computation, but early-layer feature maps can be significantly larger than later-stage representations, increasing communication time. Dynamic scheduling finds the sweet spot per device.
  • Relative Profiling Invariance: Even if client CPU availability fluctuates wildly, the ratio of runtime between Tier $A$ and Tier $B$ on the same hardware is virtually constant. This allows the server to predict latencies across unvisited tiers with minimal profiling overhead.

Key Results

  • Drastic Speedups: When training ResNet-110 on I.I.D. CIFAR-10 to 80% accuracy, DTFL finished in 4,816 seconds, compared to 24,471s for FedAvg (an ~80% time reduction), 19,299s for FedYogi, 39,676s for FedGKT, and 67,265s for SplitFed.
  • Consistent Across Distributions: On non-I.I.D. CIFAR-100 (target 50% accuracy), DTFL took 9,874s versus FedAvg’s 66,317s and SplitFed’s 183,122s.
  • Scalability: When scaling from 20 to 200 clients, DTFL maintained steady throughput and low round latency.
  • Privacy Compatibility: Incorporating privacy defenses such as distance correlation regularization (decorrelating raw data from transmitted feature maps) or patch shuffling caused negligible accuracy degradation (e.g., dropping from 87.1% to 85.4%–86.8% on CIFAR-10).

Limitations and Future Work

  • Structural Split Constraints: Models cannot be chopped arbitrarily at any layer; split points must respect architectural block boundaries (e.g., ResNet residual modules). Thus, the number of practical tiers is limited.
  • Server Computational Load: Slower clients shift more layers to the server. If many clients stall simultaneously, server-side resource contention could become a new bottleneck.
  • Auxiliary Head Expressiveness: Local greedy updates can experience performance degradation compared to end-to-end backpropagation on extremely deep, complex architectures or tasks requiring fine-grained spatial representations.
  • Future Work: Extending dynamic tiering to Large Language Models (LLMs) such as transformer architectures (e.g., BERT), as noted conceptually in the paper.

Summary updated

Abstract

Federated learning (FL) enables collaboratively training a model while keeping the training data decentralized and private. However, one significant impediment to training a model using FL, especially large models, is the resource constraints of devices with heterogeneous computation and communication capacities as well as varying task sizes. Such heterogeneity would render significant variations in the training time of clients, resulting in a longer overall training time as well as a waste of resources in faster clients. To tackle these heterogeneity issues, we propose the Dynamic Tiering-based Federated Learning (DTFL) system where slower clients dynamically offload part of the model to the server to alleviate resource constrains and speed up training. By leveraging the concept of Split Learning, DTFL offloads different portions of the global model to clients in different tiers and enables each client to update the models in parallel via local-loss-based training. This helps reduce the computation and communication demand on resource-constrained devices and thus mitigates the straggler problem. DTFL introduces a dynamic tier scheduler that uses tier profiling to estimate the expected training time of each client, based on their historical training time, communication speed, and dataset size. The dynamic tier scheduler assigns clients to suitable tiers to minimize the overall training time in each round. We first theoretically prove the convergence properties of DTFL. We then train large models (ResNet-56 and ResNet-110) on popular image datasets (CIFAR-10, CIFAR-100, CINIC-10, and HAM10000) under both IID and non-IID systems. Extensive experimental results show that compared with state-of-the-art FL methods, DTFL can significantly reduce the training time while maintaining model accuracy.
BibTeX
@misc{mohammadabadi2023speed,
  author = {Seyed Mahmoud Sajjadi Mohammadabadi and Syed Zawad and Feng Yan and Lei Yang},
  title = {{Speed Up Federated Learning in Heterogeneous Environment: A Dynamic Tiering Approach}},
  year = {2023},
  eprint = {2312.05642},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.05642},
}

Loading related content...

Figure