WHALE-FL: Wireless and Heterogeneity Aware Latency Efficient Federated Learning over Mobile Devices via Adaptive Subnetwork Scheduling

Huai-An Su, Jiaxiang Geng, Liang Li, Xiaoqi Qin, Yanzhao Hou, Hao Wang, Xin Fu, Miao Pan

TL;DR

This paper introduces WHALE-FL, a novel federated learning approach for mobile devices that dynamically adjusts the size of subnetworks trained by each device. By considering both device capabilities and the ongoing training progress, WHALE-FL significantly speeds up training without compromising accuracy, addressing a key bottleneck in real-world federated learning deployments.

What Problem Does This Paper Address?

Federated Learning (FL) holds immense promise for training models on decentralized data, especially on mobile devices, while preserving user privacy. However, deploying FL on a large scale with diverse mobile devices presents significant challenges. These devices vary greatly in their computing power, network connectivity (which can fluctuate), and the amount of local data they possess. Existing FL methods often struggle with this heterogeneity. Some approaches try to adapt by having devices train smaller "subnetworks" of the main model. However, these methods typically assign a fixed subnetwork size to each device based on its maximum capacity. This static approach fails to account for the dynamic nature of mobile device performance (e.g., temporary network congestion or background tasks) and the evolving needs of the FL training process itself. This can lead to devices becoming "stragglers," slowing down the entire training process and causing significant latency. WHALE-FL aims to solve this by introducing an adaptive subnetwork scheduling mechanism.

Methods

The core innovation of WHALE-FL lies in its adaptive subnetwork scheduling. Instead of a fixed assignment, WHALE-FL empowers each mobile device to dynamically select the size of the subnetwork it will train in each communication round. This decision is guided by a novel subnetwork selection utility function. This function cleverly balances two key aspects:

  • System Efficiency: This considers the device's current computing and communication capabilities, factoring in transmission and computing delays. A higher system efficiency utility suggests the device can handle a larger subnetwork.
  • Training Efficiency: This leverages Fisher Information (specifically, a federated version called FedFIM) to gauge the importance of local training contributions at different stages of the FL process. Higher training efficiency utility indicates a need for more impactful local updates, potentially requiring a larger subnetwork.

The utility function combines these two factors using a developer-specified trade-off parameter ($\beta$). The resulting utility value is then normalized and used to select an appropriate subnetwork size from a predefined set of options. This adaptive approach allows devices to adjust their subnetwork size based on their immediate conditions and the overall training progress, aiming to optimize for both speed and accuracy.

Datasets

The paper evaluates WHALE-FL on three distinct FL tasks using standard datasets:

  • Image Classification:
    • CNN on MNIST (Deng, 2012)
    • ResNet18 on CIFAR10 (He et al., 2015)
  • Human Activity Recognition:
    • CNN on the HAR dataset (Gupta et al., 2022)
  • Language Modeling:
    • Transformer on the WikiText2 dataset (Devlin et al., 2019)

The experiments utilize a balanced non-IID data partition strategy, where each device is assigned a subset of classes, simulating a more realistic scenario.

Key Insights and Ideas

  • Dynamic is Key: The most significant insight is that static subnetwork assignments are insufficient for heterogeneous mobile FL. Adapting to real-time system dynamics (network, compute) and FL training dynamics (critical learning periods) is crucial for efficiency.
  • Fisher Information as a Training Progress Indicator: Using Fisher Information to estimate the "critical learning periods" and the diminishing returns of local updates is a smart way to inform subnetwork size selection for training efficiency.
  • Balancing Act: The core of WHALE-FL is the intelligent trade-off between system efficiency (how fast can I train now) and training efficiency (how much will my training help the global model). This balance shifts throughout the training process.
  • Subnetwork Size as a Lever: The paper demonstrates that dynamically adjusting subnetwork size is a powerful lever to accelerate FL, rather than just relying on faster hardware or communication.

Key Results

WHALE-FL consistently demonstrated significant improvements across various FL tasks:

  • Speedup: WHALE-FL achieved substantial speedups compared to FedAvg (all devices train full models) and HeteroFL (static subnetwork assignment). For example, it accelerated FL training by approximately 1.5x to 2.1x compared to FedAvg and 1.06x to 1.74x compared to HeteroFL across different tasks.
  • Accuracy: Crucially, these speedups were achieved without sacrificing learning accuracy. In many cases, WHALE-FL even showed slight improvements in final accuracy compared to baseline methods.
  • Adaptive Behavior: Experiments showed that WHALE-FL effectively adjusted subnetwork sizes based on local Fisher Information and device capabilities, aligning with the paper's theoretical underpinnings.
  • Robustness: The approach showed good performance even under varying degrees of data heterogeneity.

Limitations and Future Work

  • Hyperparameter Tuning: While WHALE-FL offers flexibility, the performance is influenced by hyperparameters like $\beta$ (trade-off between system and training efficiency), $D$ (window size for averaging Fisher information), and $U_{th}$ (utility threshold for full model adoption). The paper provides sensitivity analysis, but optimal tuning might still require effort.
  • Complexity of Fisher Information: Calculating Fisher Information can be computationally intensive, although the paper uses a federated approach to mitigate this.
  • Future Work: The authors suggest further exploration into more sophisticated adaptive scheduling policies and potentially integrating other system dynamics beyond just computing and communication. They also mention theoretical convergence analysis for WHALE-FL, indicating ongoing research into its theoretical guarantees.

Summary updated

Abstract

As a popular distributed learning paradigm, federated learning (FL) over mobile devices fosters numerous applications, while their practical deployment is hindered by participating devices' computing and communication heterogeneity. Some pioneering research efforts proposed to extract subnetworks from the global model, and assign as large a subnetwork as possible to the device for local training based on its full computing capacity. Although such fixed size subnetwork assignment enables FL training over heterogeneous mobile devices, it is unaware of (i) the dynamic changes of devices' communication and computing conditions and (ii) FL training progress and its dynamic requirements of local training contributions, both of which may cause very long FL training delay. Motivated by those dynamics, in this paper, we develop a wireless and heterogeneity aware latency efficient FL (WHALE-FL) approach to accelerate FL training through adaptive subnetwork scheduling. Instead of sticking to the fixed size subnetwork, WHALE-FL introduces a novel subnetwork selection utility function to capture device and FL training dynamics, and guides the mobile device to adaptively select the subnetwork size for local training based on (a) its computing and communication capacity, (b) its dynamic computing and/or communication conditions, and (c) FL training status and its corresponding requirements for local training contributions. Our evaluation shows that, compared with peer designs, WHALE-FL effectively accelerates FL training without sacrificing learning accuracy.
BibTeX
@inproceedings{su2025whale,
  author = {Huai-An Su and Jiaxiang Geng and Liang Li and Xiaoqi Qin and Yanzhao Hou and Hao Wang and Xin Fu and Miao Pan},
  title = {{WHALE-FL: Wireless and Heterogeneity Aware Latency Efficient Federated Learning over Mobile Devices via Adaptive Subnetwork Scheduling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i19.34272},
}
Figure