CCFL: Computationally Customized Federated Learning

Hao Zhang, Tingting Wu, Siyao Cheng, Jie Liu

TL;DR

CC-FedAvg introduces a novel "local model estimation" strategy that allows resource-constrained IoT devices to skip expensive training iterations in federated learning without hurting global model accuracy. By predicting local updates using historical trajectories, it maintains the same convergence rate as standard FedAvg while drastically lowering computational overhead.

What Problem Does This Paper Address?

Standard Federated Learning (FL) algorithms, like the ubiquitous FedAvg, operate under a flawed assumption: that all participating devices (clients) possess uniform computational capacity. In real-world Internet of Things (IoT) networks, devices face highly heterogeneous conditions, such as restricted energy budgets, thermal throttling, or competing background tasks.

When low-resource devices cannot complete the required local training iterations in time, they become stragglers or drop out entirely. Traditional FL handles this poorly: ignoring these devices introduces severe data bias (over-representing resource-rich devices), while simply reusing their stale models from previous rounds degrades the global model's accuracy. The authors set out to find a way for resource-constrained clients to participate meaningfully in every round without incurring heavy on-device computational costs.

Methods

The authors propose Computationally Customized Federated Averaging (CC-FedAvg). Instead of forcing every selected client to run resource-intensive Stochastic Gradient Descent (SGD) iterations, CC-FedAvg allows clients to probabilistically choose between traditional local training and a zero-overhead local estimation strategy based on their real-time resource budgets.

Fig. 1. Illustration of CC-FedAvg solving computation heterogeneity in FL training. “Local training” refers to the traditional iterative method such as SGD to obtain local models, which is computationally expensive. “Local estimation” refers to estimating the local model from historical information with negligible computational overhead. (a) Devices schedule to train or estimate local models in advance based on their energy budgets. Devices with lower energy reserves perform local estimation with higher probability. (b) Devices decide whether to execute local training or local estimation based on the tasks in time. Devices with higher load perform local estimation with higher probability.
  • The Estimation Strategy (Strategy 3): If a client lacks the resources to train in a given round, it "guesses" its current update vector ($\Delta_t^i$) by simply reusing its update vector from the previous round ($\Delta_{t-1}^i$).
  • Probabilistic Customization: Each client $i$ performs full local training with a probability $p_i$ (proportional to its available resources) and performs local estimation with probability $1-p_i$.
  • Server-Side Offloading (Algorithms 2 & 3): To minimize client-side storage and communication overhead, the authors design variants where the historical update vectors are backed up on the server. When a client needs to skip a round, it only transmits a single "skip" bit, and the server applies the historical estimation on its behalf.

Datasets

The authors evaluated CC-FedAvg across three standard computer vision benchmarks with varying model architectures:

  • CIFAR-10: Evaluated using both a shallow CNN and a ResNet-18 architecture.
  • Fashion-MNIST (FMNIST): Evaluated using a Multi-Layer Perceptron (MLP).
  • CIFAR-100: Evaluated using a ResNet-18 model with Group Normalization.

Data heterogeneity was explicitly controlled using a hyperparameter $\gamma$ to simulate varying degrees of non-IID (Independent and Identically Distributed) data splits across clients.

Key Insights and Ideas

  • History Repeats Itself (Locally): The core intuition behind CC-FedAvg is that local model updates change continuously across rounds. Reusing the direction and magnitude of the previous update provides a remarkably accurate approximation of a fully trained model, especially during the early-to-mid stages of training. This vastly outperforms the naive approach of using stale model weights.
  • No Convergence Penalty: The authors provide a rigorous theoretical proof showing that CC-FedAvg achieves the exact same order of convergence speed—$O(1/\sqrt{NKT})$—as vanilla FedAvg operating under full, unconstrained client participation.
  • A General Efficiency Booster: CC-FedAvg isn't just for resource-constrained networks. If applied uniformly to all clients (setting the resource ratio $r=1$), it acts as a computation-efficient extension of standard FL. It can slash total training computation to 1/4th of standard levels while preserving final model accuracy.

Key Results

  • Comparable Accuracy to Full Training: On a highly non-IID CIFAR-10 setup, CC-FedAvg achieved a top-1 test accuracy of 56.28%, nearly matching the unconstrained FedAvg baseline (56.82%)—despite the fact that 75% of the clients were resource-constrained and some only trained 1/8th of the time.
  • Robustness Against Dropouts: Under identical resource constraints, standard FedAvg with client dropouts saw its accuracy plunge to between 25.35% and 40.4%. CC-FedAvg easily outperformed other baseline estimation strategies across both "round-robin" and "ad-hoc" task schedules.
  • Outperforming FedNova: CC-FedAvg consistently beat FedNova (a popular framework for handling variable local updates), particularly when the number of local iterations ($K$) was small. FedNova's performance degrades when local training is highly inadequate, whereas CC-FedAvg remains stable.

Limitations and Future Work

While highly effective, the authors note that the accuracy of reusing the previous update vector degrades slightly in the very late stages of training as the global model begins to stabilize and fine-tune. To combat this, they briefly introduce a hybrid variant, CC-FedAvg(c), which switches from update estimation to stale model reuse after a threshold number of rounds ($\tau$).

Additionally, if the data distribution is highly skewed and correlated with resource constraints (e.g., if all clients holding a specific class of data happen to be the lowest-power devices), model performance degrades more noticeably. Developing more adaptive, data-aware local estimation strategies remains an open direction for future work.

Summary updated

Abstract

Federated learning (FL) is an emerging paradigm to train model with distributed data from numerous Internet of Things (IoT) devices. It inherently assumes a uniform capacity among participants. However, due to different conditions such as differing energy budgets or executing parallel unrelated tasks, participants have diverse computational resources in practice. Participants with insufficient computation budgets must plan for the use of restricted computational resources appropriately, otherwise they would be unable to complete the entire training procedure, resulting in model performance decline. To address this issue, we propose a strategy for estimating local models without computationally intensive iterations. Based on it, we propose Computationally Customized Federated Averaging (CC-FedAvg), which allows participants to determine whether to perform traditional local training or model estimation in each round based on their current computational budgets. Both theoretical analysis and exhaustive experiments indicate that CC-FedAvg has the same convergence rate and comparable performance as FedAvg without resource constraints. Furthermore, CC-FedAvg can be viewed as a computation-efficient version of FedAvg that retains model performance while considerably lowering computation overhead.
BibTeX
@article{zhang2024ccfl,
  author = {Hao Zhang and Tingting Wu and Siyao Cheng and Jie Liu},
  title = {{CCFL: Computationally Customized Federated Learning}},
  journal = {IEEE Internet of Things Journal},
  year = {2024},
  doi = {10.1109/jiot.2023.3300080},
}
Figure