TL;DR
While Federated Learning (FL) protects raw data from direct exposure, Large Language Models (LLMs) trained in FL can still inadvertently memorize and leak sensitive training text via targeted prompting. This paper demonstrates that replacing full fine-tuning with Low-Rank Adaptation (LoRA) drastically mitigates this unintended memorization—reducing it by up to a factor of 10—across multiple domains and model sizes (1B to 70B parameters) with virtually no cost to downstream performance.
What Problem Does This Paper Address?
Federated Learning is widely celebrated for enabling collaborative model training without centralizing sensitive user data. However, LLMs have a well-documented habit of verbatim memorization. In an FL setting, an adversarial or "honest-but-curious" client can exploit this by prompting the shared global model with specific prefixes to reconstruct the private training data of other participants.
While Differential Privacy (DP) offers theoretical guarantees against such leaks, applying it to LLMs often results in severe performance degradation and introduces complex questions about how to define privacy boundaries in natural language. The authors set out to find a practical, empirical alternative to reduce unintended memorization during fine-tuning without destroying the model's utility.
Methods
The authors comprehensively evaluate the privacy benefits of Parameter-Efficient Fine-Tuning (PEFT), specifically focusing on Low-Rank Adaptation (LoRA), compared to full parameter fine-tuning.
Their core methodology involves:
- Cross-Silo FL Simulation: A realistic 3-client heterogeneous setup using Federated Averaging (FedAvg), mimicking a collaborative scenario like hospitals training a joint medical model.
- Canary Injection: Injecting real-world sensitive text sequences into the training data, both with and without duplication (e.g., 10x duplication), to measure how easily the model overfits and memorizes them.
- Quantifying Memorization: Prompting the fine-tuned models with varying prefix lengths (10 to 500 tokens) and using greedy decoding to generate a 50-token suffix. Memorization is evaluated using exact token matching, BLEU scores (where a score > 0.75 signifies approximate reproduction), and BERTScore.
- Defense Synergy Exploration: Testing how LoRA interacts with other privacy-preserving strategies, including gradient clipping, Goldfish loss, NEFTune, and cryptographic Secure Aggregation (FHE + SMPC).
Datasets
The primary experiments focus on the medical domain using three QA datasets:
- MedMCQA
- PubMedQA
- Medical Meadow Flashcards
To measure privacy leaks, these datasets were augmented with sensitive clinical notes from the 2014 i2b2/UTHealth de-identification corpus. To ensure their findings generalize, the authors also evaluated models on high-risk datasets from other domains: Ai2’s Multi-LexSum (Law) and ConvFinQA (Finance).
Key Insights and Ideas
- LoRA as an Implicit Regularizer: LoRA restricts weight updates to a low-rank subspace ($\Delta W = BA$). The authors hypothesize that this constraint prevents "benign overfitting"—a phenomenon where a model overfits to the noise or specific wording of training data without harming its general performance. By ignoring minor singular vectors in the gradient updates, LoRA naturally drops the capacity to memorize specific text strings.
- Synergy with Federated Learning: Interestingly, the decentralized nature of FL (FedAvg combined with non-IID data distributions) inherently reduces memorization compared to Centralized Learning (CL). When combined with LoRA, this reduction is amplified, creating a highly effective defense strategy.
- The Rank Tradeoff: The rank ($r$) of the LoRA matrices acts as a direct knob for privacy. Lower ranks yield significantly less memorization but can eventually degrade accuracy if set too low.
Key Results
- Up to 10x Memorization Reduction: Across all tested model families (Llama-2, Llama-3, and Mistral) and sizes (up to 70B), LoRA consistently achieved lower memorization scores than full fine-tuning.
- Negligible Performance Cost: As shown in the downstream accuracy benchmarks, the privacy gains from LoRA do not come at the expense of utility; LoRA models perform on par with, or occasionally better than, fully fine-tuned models.
- Rank Impact: When tuning a Llama 3.2 3B model, a LoRA rank of 4 resulted in virtually zero memorization (BLEU score of 0.0198 on 10x duplicated data), whereas a rank of 1024 behaved similarly to full fine-tuning, memorizing nearly 50% of the duplicated records.
- Synergistic Defenses: Combining LoRA with Goldfish loss (which randomly excludes tokens from the loss calculation) or aggressive gradient clipping pushed memorization rates even lower, providing a robust defense-in-depth framework.
Limitations and Future Work
- Scale of FL Setup: The study focuses on a cross-silo setting with a small number of clients (3). Whether these exact dynamics hold true in massive cross-device federated settings (with millions of clients holding tiny fractions of data) remains an open question.
- Theoretical Gaps: While the empirical evidence is strong, a rigorous mathematical foundation explaining why low-rank updates specifically target memorization while preserving generalization is still developing.
- Not a Perfect Silver Bullet: LoRA drastically reduces, but does not completely eliminate, memorization. The authors caution that for absolute privacy guarantees, sensitive data must still be scrubbed before training, and LoRA should be viewed as a powerful mitigation layer rather than a total cure.
Summary updated