Introduction
The deployment of Large Language Models (LLMs) on mobile and edge devices is a critical frontier for personalized AI. However, training and fine-tuning these models—often containing billions of parameters—presents a "three-headed monster" of challenges in wireless federated learning (FL) environments. First, data heterogeneity means that non-IID (not identically and independently distributed) user data often leads to poor performance for a generic global model. Second, resource constraints on mobile devices limit the ability to perform full-scale training due to memory and compute bottlenecks. Finally, communication overhead in wireless channels creates significant latency and battery drain when synchronizing massive parameter sets.
To address these hurdles, the Personalized Wireless Federated Fine-tuning (PWFF) framework is proposed. PWFF enables the deployment of user-centric LLMs by combining parameter-efficient fine-tuning with a multi-objective reinforcement learning approach, specifically designed for the constraints of wireless networks.
The PWFF Framework
The PWFF framework introduces three primary innovations to make federated LLM fine-tuning viable on resource-constrained devices.
Global Partial Aggregation
Instead of updating the entire model, PWFF utilizes Parameter-Efficient Fine-Tuning (PEFT). The architecture inserts lightweight Adapters and LoRA (Low-Rank Adaptation) modules into the LLM. A key strategic choice in PWFF is the split aggregation: only the Adapter parameters are transmitted to the server for global aggregation to share general linguistic knowledge. In contrast, the LoRA parameters remain local to the device, capturing user-specific preferences and maintaining a personalized "flavor" without incurring additional communication costs.
Personalized Reward Models and Alignment
To align LLMs with diverse human values, PWFF employs Reinforcement Learning from Human Feedback (RLHF). It defines two distinct reward models: one for helpfulness (accuracy and relevance) and one for harmlessness (legality and ethics). This allows for a "Personalization as a Pareto Front" approach, where users can weight these rewards differently based on their specific needs—for instance, prioritizing safety for a child's device versus helpfulness for a research tool.
Workflow: From Tuning to Alignment
The PWFF process is structured into three distinct federated steps, as illustrated in the following workflow:
- Federated Instruction Tuning: Clients perform local instruction tuning using PEFT. Adapters are aggregated globally while LoRA modules stay local.
- Federated Reward Learning: Local preference data is used to train personalized reward models. These models are then aggregated to form a robust baseline for alignment.
- Federated Alignment Tuning: The framework utilizes a personalized loss function for the PPO (Proximal Policy Optimization) algorithm. It employs two reward models and two "critic" models to ensure that local policy updates remain stable while balancing the competing objectives of helpfulness and safety.
Key Insights and Results
Wireless-Aware Design
Unlike traditional AI frameworks, PWFF is explicitly designed for 6G applications. It factors in the Rayleigh channel model, Signal-to-Noise Ratio (SNR), and transmission power. This wireless-aware approach ensures that the model remains performant even under fluctuating signal conditions.
Performance Metrics
- Superior Accuracy: PWFF achieved higher accuracy in instruction-following tasks compared to standard benchmarks like FedLora, demonstrating that its personalized structure handles non-IID data more effectively.
- Energy Efficiency: The partial aggregation strategy significantly reduced transmission energy consumption by 43% compared to standard Personalized Federated Learning (PFL) approaches.
- Effective Personalization: Simulations confirmed that local models successfully obtained higher rewards when tuned to specific user-defined weights for helpfulness and harmlessness, effectively navigating the Pareto front of model alignment.
Conclusion
The PWFF framework represents a significant step toward private, efficient, and personalized AI on the edge. By strategically splitting parameter updates and employing multi-objective RLHF, it overcomes the primary bottlenecks of wireless federated learning. Future work will focus on addressing wireless instability through asynchronous aggregation and exploring advanced regularization techniques to prevent overfitting on small local datasets.
Summary updated