Abstract
Training Large Language Models (LLMs) on private, siloed data presents significant privacy and hardware challenges. We present FL-GLM, a novel federated learning framework specifically engineered for LLMs. By combining strategic model splitting with parallel acceleration techniques, FL-GLM achieves performance parity with centralized training while providing robust protection against data leakage. Our approach effectively mitigates the security risks of standard split learning and the hardware bottlenecks of traditional federated learning.
Introduction
Large Language Models thrive on high-quality data, much of which is "private"—stored in isolated silos like medical records or personal mobile devices. Training on this data is a privacy minefield. Traditional Federated Learning (FL), like FedAvg, requires clients to download and train the entire model locally, which is impossible for most consumer hardware given the billions of parameters in modern LLMs.
An alternative, Split Learning (SL), offloads most of the work to a powerful server, but it has two major flaws: security risks where gradients can be reverse-engineered to reconstruct private text, and efficiency bottlenecks due to its serial nature. FL-GLM addresses these by rethinking the model split and introducing parallel processing.
The FL-GLM Framework
FL-GLM introduces three core innovations to balance security and efficiency:
Strategic Model Splitting
Unlike standard split learning that only keeps the very first and last layers on the client, FL-GLM places the input block (Embedding + Block 0) and the output block (Block N-1 + Linear layer) on the client. By keeping the first transformer block local, the gradients sent to the server are "smashed" through complex non-linear transformations, making it nearly impossible for a malicious server to reconstruct the input data.
Parallel Acceleration and Security
To fix the "one-at-a-time" bottleneck of serial split learning, FL-GLM proposes two modes:
- Client-batch: The server stacks "smashed" data from multiple clients into a single large batch, utilizing GPU parallelism.
- Server-hierarchical: For massive clusters, a central server manages multiple sub-servers that handle clients in parallel, aggregating updates periodically.
To prevent "peer-client" attacks (where one client intercepts another's data), FL-GLM uses RSA key encryption for all communications between the client and the server.
Experimental Evaluation
The framework was evaluated using ChatGLM-6B and Llama2-7B across several benchmarks, including SuperGLUE for NLU tasks, CNN/DailyMail for summarization, and the Huatuo-26M dataset for specialized Chinese medical Q&A.
Performance and Efficiency
FL-GLM achieved results nearly identical to centralized training. For example, on the SuperGLUE benchmark, FL-GLM scored an average of 79.1, compared to 79.6 for the centralized ChatGLM-6B. Parallel strategies reduced training time significantly; in a test with 8 clients, the client-batch method was roughly 7.5x faster than the serial approach, representing a 48% reduction in total training time.
Security Robustness
In simulated feature-space hijacking attacks (FSHA), an adversary trying to reconstruct data from a client with only an embedding layer achieved a ROUGE-1 score of 33.29. When the client held "Block 0" (as in FL-GLM), the attack score plummeted to 0.14, effectively rendering the attack useless.
Conclusion
FL-GLM demonstrates that it is possible to train massive LLMs on private data without compromising security or performance. While a privacy-utility trade-off remains, the "Block 0" defense provides a significant security boost with minimal overhead. Future work will focus on optimizing communication for slower network connections and integrating Differential Privacy for even stronger mathematical guarantees against data leakage.
Summary updated