Personalized Wireless Federated Learning for Large Language Models

Feibo Jiang, Li Dong, Siwei Tu, Yubo Peng, Kezhi Wang, Kun Yang, Cunhua Pan, Dusit Niyato

Introduction

The deployment of Large Language Models (LLMs) on mobile and edge devices is a critical frontier for personalized AI. However, training and fine-tuning these models—often containing billions of parameters—presents a "three-headed monster" of challenges in wireless federated learning (FL) environments. First, data heterogeneity means that non-IID (not identically and independently distributed) user data often leads to poor performance for a generic global model. Second, resource constraints on mobile devices limit the ability to perform full-scale training due to memory and compute bottlenecks. Finally, communication overhead in wireless channels creates significant latency and battery drain when synchronizing massive parameter sets.

To address these hurdles, the Personalized Wireless Federated Fine-tuning (PWFF) framework is proposed. PWFF enables the deployment of user-centric LLMs by combining parameter-efficient fine-tuning with a multi-objective reinforcement learning approach, specifically designed for the constraints of wireless networks.

The PWFF Framework

The PWFF framework introduces three primary innovations to make federated LLM fine-tuning viable on resource-constrained devices.

Global Partial Aggregation

Instead of updating the entire model, PWFF utilizes Parameter-Efficient Fine-Tuning (PEFT). The architecture inserts lightweight Adapters and LoRA (Low-Rank Adaptation) modules into the LLM. A key strategic choice in PWFF is the split aggregation: only the Adapter parameters are transmitted to the server for global aggregation to share general linguistic knowledge. In contrast, the LoRA parameters remain local to the device, capturing user-specific preferences and maintaining a personalized "flavor" without incurring additional communication costs.

Overview of the PWFF framework showing Global Partial Aggregation and Local Multi-objective Alignment.

Personalized Reward Models and Alignment

To align LLMs with diverse human values, PWFF employs Reinforcement Learning from Human Feedback (RLHF). It defines two distinct reward models: one for helpfulness (accuracy and relevance) and one for harmlessness (legality and ethics). This allows for a "Personalization as a Pareto Front" approach, where users can weight these rewards differently based on their specific needs—for instance, prioritizing safety for a child's device versus helpfulness for a research tool.

Workflow: From Tuning to Alignment

The PWFF process is structured into three distinct federated steps, as illustrated in the following workflow:

  1. Federated Instruction Tuning: Clients perform local instruction tuning using PEFT. Adapters are aggregated globally while LoRA modules stay local.
  2. Federated Reward Learning: Local preference data is used to train personalized reward models. These models are then aggregated to form a robust baseline for alignment.
  3. Federated Alignment Tuning: The framework utilizes a personalized loss function for the PPO (Proximal Policy Optimization) algorithm. It employs two reward models and two "critic" models to ensure that local policy updates remain stable while balancing the competing objectives of helpfulness and safety.
The three-step federated process: Instruction Tuning, Reward Learning, and Alignment Tuning.

Key Insights and Results

Wireless-Aware Design

Unlike traditional AI frameworks, PWFF is explicitly designed for 6G applications. It factors in the Rayleigh channel model, Signal-to-Noise Ratio (SNR), and transmission power. This wireless-aware approach ensures that the model remains performant even under fluctuating signal conditions.

Performance Metrics

  • Superior Accuracy: PWFF achieved higher accuracy in instruction-following tasks compared to standard benchmarks like FedLora, demonstrating that its personalized structure handles non-IID data more effectively.
  • Energy Efficiency: The partial aggregation strategy significantly reduced transmission energy consumption by 43% compared to standard Personalized Federated Learning (PFL) approaches.
  • Effective Personalization: Simulations confirmed that local models successfully obtained higher rewards when tuned to specific user-defined weights for helpfulness and harmlessness, effectively navigating the Pareto front of model alignment.

Conclusion

The PWFF framework represents a significant step toward private, efficient, and personalized AI on the edge. By strategically splitting parameter updates and employing multi-objective RLHF, it overcomes the primary bottlenecks of wireless federated learning. Future work will focus on addressing wireless instability through asynchronous aggregation and exploring advanced regularization techniques to prevent overfitting on small local datasets.

Summary updated

Abstract

Large Language Models (LLMs) have revolutionized natural language processing tasks. However, their deployment in wireless networks still face challenges, i.e., a lack of privacy and security protection mechanisms. Federated Learning (FL) has emerged as a promising approach to address these challenges. Yet, it suffers from issues including inefficient handling with big and heterogeneous data, resource-intensive training, and high communication overhead. To tackle these issues, we first compare different learning stages and their features of LLMs in wireless networks. Next, we introduce two personalized wireless federated fine-tuning methods with low communication overhead, i.e., (1) Personalized Federated Instruction Tuning (PFIT), which employs reinforcement learning to fine-tune local LLMs with diverse reward models to achieve personalization; (2) Personalized Federated Task Tuning (PFTT), which can leverage global adapters and local Low-Rank Adaptations (LoRA) to collaboratively fine-tune local LLMs, where the local LoRAs can be applied to achieve personalization without aggregation. Finally, we perform simulations to demonstrate the effectiveness of the proposed two methods and comprehensively discuss open issues.
BibTeX
@misc{jiang2024personalized,
  author = {Feibo Jiang and Li Dong and Siwei Tu and Yubo Peng and Kezhi Wang and Kun Yang and Cunhua Pan and Dusit Niyato},
  title = {{Personalized Wireless Federated Learning for Large Language Models}},
  year = {2024},
  eprint = {2404.13238},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.13238},
}
Figure