Low-Latency Federated Fine-Tuning for Large Language Models Over Wireless Networks

Zhiwen Pang, Kang Wei, Long Shi, Zhe Wang, Jun Li, Feng Shu

TL;DR

This paper introduces JCPBA, a novel framework for federated fine-tuning of Large Language Models (LLMs) over wireless networks. JCPBA dynamically optimizes client-specific pruning and bandwidth allocation to minimize fine-tuning latency, significantly outperforming existing methods in speed and efficiency while maintaining model performance.

What Problem Does This Paper Address?

Federated learning (FL) offers a privacy-preserving way to fine-tune LLMs using distributed data. However, the massive size of LLMs presents significant challenges for FL, especially in resource-constrained environments like edge devices connected via wireless networks. Clients often have heterogeneous computing power and unreliable wireless channels, leading to "stragglers" and large variations in training latency. Existing approaches that try to improve efficiency often struggle to minimize this per-round latency, hindering practical deployment. This paper aims to tackle this challenge by developing a federated LLM fine-tuning approach that specifically targets low latency in such heterogeneous wireless settings.

Methods

The core of this work is the Joint Client-Specific Pruning and Bandwidth Allocation (JCPBA) framework. JCPBA builds upon the "emulator-adapter" design, where the LLM is split into a fixed emulator (representing general knowledge) and an updatable adapter (for task-specific learning).

Here's how JCPBA works:

  • Model Partitioning and Pruning: The server partitions the pre-trained LLM into an emulator and an adapter. Crucially, the server applies structured pruning to the emulator before sending it to clients. This pruning is client-specific, with the server assigning a unique pruning rate ($\beta_k$) to each client based on their capabilities. This reduces the computational load on the emulator.
  • Client-Specific Bandwidth Allocation: Alongside pruning, the server also dynamically allocates bandwidth ($B_k$) to each client. This allocation considers the client's wireless channel conditions and aims to optimize communication efficiency.
  • Local Fine-tuning: Clients then fine-tune only the adapter using their local, private data, keeping the pruned emulator fixed.
  • Parameter Upload: Clients upload only the adapter updates to the server.
  • Aggregation and Server-Side Update: The server aggregates these updates and performs a server-side fine-tuning of the emulator using public data to maintain its general capabilities.

The problem of minimizing the maximum per-round latency across all clients is formulated as an optimization problem, jointly optimizing pruning rates and bandwidth allocations. This problem is solved using a Block Coordinate Descent (BCD) method, which iteratively optimizes pruning rates and bandwidth allocations separately.

Datasets

The experiments were conducted on two representative downstream tasks:

  • Yahoo Answers: Used for text classification.
  • GSM8K: Used for mathematical reasoning.

The pre-trained model used was GPT-2 Medium.

Key Insights and Ideas

  • Dynamic Client-Specific Optimization is Key: The paper highlights that a one-size-fits-all approach to pruning and bandwidth allocation is insufficient for heterogeneous environments. JCPBA's strength lies in its ability to tailor these parameters to each client's unique computational and communication capabilities.
  • Pruning the Emulator for Latency Reduction: By pruning the emulator (which is fixed during client-side fine-tuning), the computational cost on the client is significantly reduced. This is a novel way to leverage pruning for federated LLM fine-tuning latency.
  • Joint Optimization Pays Off: The simultaneous optimization of pruning and bandwidth allocation is crucial. Optimizing one without considering the other can lead to suboptimal results, as computation and communication are intertwined in the overall latency.
  • Mitigating Stragglers: The dynamic adaptation of JCPBA effectively mitigates the impact of "straggler" clients (those with slower computation or poorer network connections), leading to more stable and predictable fine-tuning times across the entire network.

Key Results

  • Significant Latency Reduction: JCPBA reduces wall-clock fine-tuning time by at least 40% compared to state-of-the-art baselines.
  • Comparable or Better Model Performance: The framework achieves equal or lower test loss than baselines, demonstrating that efficiency gains do not come at the cost of model accuracy.
  • Reduced Communication Overhead: Communication overhead is reduced by at least 46%.
  • Robustness to Heterogeneity: In experiments with varying levels of computation heterogeneity, JCPBA showed the smallest increase in fine-tuning latency, with its latency only rising by about 6% across different heterogeneity levels, compared to much larger increases for baselines (e.g., 176% for Split-LoRA).

Limitations and Future Work

  • Limitations: The paper focuses on a single-round latency minimization problem for simplicity. While this is a reasonable simplification, real-world federated learning involves multiple rounds, and the long-term impact of this per-round optimization might warrant further investigation. The paper also assumes a centralized server managing the FL process.
  • Future Work: The authors suggest that JCPBA's efficient and scalable solution makes it suitable for various applications of federated LLM fine-tuning over wireless networks. Future work could explore decentralized FL architectures or more complex communication patterns.

Summary updated

Abstract

Recently, federated large language models (LLMs) have drawn significant attention thanks to coupled capabilities of LLMs and federated learning (FL) that address privacy concerns in collaborative fine-tuning. However, due to large-scale parameters of LLMs, existing federated LLM fine-tuning frameworks incur significant challenges in resource-constrained clients characterized by heterogeneous computing capabilities and random wireless channels. To address this issue, we propose a joint client-specific pruning and bandwidth allocation (JCPBA) framework for federated LLMs to improve the fine-tuning efficiency over the wireless networks. Specifically, we formulate a fine-tuning latency minimization problem by jointly optimizing pruning rates and bandwidth allocations. Furthermore, we solve this optimization problem using a block coordinate descent method. Extensive experiments on the datasets of Yahoo Answers and GSM8K demonstrate that the proposed framework significantly reduces wall-clock fine-tuning time compared with state-of-the-art baselines and gains equal or lower test loss at the cost of lower computation and communication overhead.
BibTeX
@article{pang2026low,
  author = {Zhiwen Pang and Kang Wei and Long Shi and Zhe Wang and Jun Li and Feng Shu},
  title = {{Low-Latency Federated Fine-Tuning for Large Language Models Over Wireless Networks}},
  journal = {IEEE Wireless Communications Letters},
  year = {2026},
  url = {https://openreview.net/forum?id=8wr604V13G},
}
Figure