Mitigating Unintended Memorization with LoRA in Federated Learning for LLMs

Thierry Bossy, Julien Vignoud, Tahseen Rabbani, Juan R. Troncoso Pastoriza, Martin Jaggi

TL;DR

While Federated Learning (FL) protects raw data from direct exposure, Large Language Models (LLMs) trained in FL can still inadvertently memorize and leak sensitive training text via targeted prompting. This paper demonstrates that replacing full fine-tuning with Low-Rank Adaptation (LoRA) drastically mitigates this unintended memorization—reducing it by up to a factor of 10—across multiple domains and model sizes (1B to 70B parameters) with virtually no cost to downstream performance.

What Problem Does This Paper Address?

Federated Learning is widely celebrated for enabling collaborative model training without centralizing sensitive user data. However, LLMs have a well-documented habit of verbatim memorization. In an FL setting, an adversarial or "honest-but-curious" client can exploit this by prompting the shared global model with specific prefixes to reconstruct the private training data of other participants.

While Differential Privacy (DP) offers theoretical guarantees against such leaks, applying it to LLMs often results in severe performance degradation and introduces complex questions about how to define privacy boundaries in natural language. The authors set out to find a practical, empirical alternative to reduce unintended memorization during fine-tuning without destroying the model's utility.

Methods

The authors comprehensively evaluate the privacy benefits of Parameter-Efficient Fine-Tuning (PEFT), specifically focusing on Low-Rank Adaptation (LoRA), compared to full parameter fine-tuning.

Their core methodology involves:

  • Cross-Silo FL Simulation: A realistic 3-client heterogeneous setup using Federated Averaging (FedAvg), mimicking a collaborative scenario like hospitals training a joint medical model.
  • Canary Injection: Injecting real-world sensitive text sequences into the training data, both with and without duplication (e.g., 10x duplication), to measure how easily the model overfits and memorizes them.
  • Quantifying Memorization: Prompting the fine-tuned models with varying prefix lengths (10 to 500 tokens) and using greedy decoding to generate a 50-token suffix. Memorization is evaluated using exact token matching, BLEU scores (where a score > 0.75 signifies approximate reproduction), and BERTScore.
  • Defense Synergy Exploration: Testing how LoRA interacts with other privacy-preserving strategies, including gradient clipping, Goldfish loss, NEFTune, and cryptographic Secure Aggregation (FHE + SMPC).

Datasets

The primary experiments focus on the medical domain using three QA datasets:

  • MedMCQA
  • PubMedQA
  • Medical Meadow Flashcards

To measure privacy leaks, these datasets were augmented with sensitive clinical notes from the 2014 i2b2/UTHealth de-identification corpus. To ensure their findings generalize, the authors also evaluated models on high-risk datasets from other domains: Ai2’s Multi-LexSum (Law) and ConvFinQA (Finance).

Key Insights and Ideas

  • LoRA as an Implicit Regularizer: LoRA restricts weight updates to a low-rank subspace ($\Delta W = BA$). The authors hypothesize that this constraint prevents "benign overfitting"—a phenomenon where a model overfits to the noise or specific wording of training data without harming its general performance. By ignoring minor singular vectors in the gradient updates, LoRA naturally drops the capacity to memorize specific text strings.
  • Synergy with Federated Learning: Interestingly, the decentralized nature of FL (FedAvg combined with non-IID data distributions) inherently reduces memorization compared to Centralized Learning (CL). When combined with LoRA, this reduction is amplified, creating a highly effective defense strategy.
  • The Rank Tradeoff: The rank ($r$) of the LoRA matrices acts as a direct knob for privacy. Lower ranks yield significantly less memorization but can eventually degrade accuracy if set too low.

Key Results

  • Up to 10x Memorization Reduction: Across all tested model families (Llama-2, Llama-3, and Mistral) and sizes (up to 70B), LoRA consistently achieved lower memorization scores than full fine-tuning.
LoRA vs. Full Fine-Tuning in Federated Learning
  • Negligible Performance Cost: As shown in the downstream accuracy benchmarks, the privacy gains from LoRA do not come at the expense of utility; LoRA models perform on par with, or occasionally better than, fully fine-tuned models.
Accuracy vs. Privacy Tradeoff Across Fine-Tuning Steps
  • Rank Impact: When tuning a Llama 3.2 3B model, a LoRA rank of 4 resulted in virtually zero memorization (BLEU score of 0.0198 on 10x duplicated data), whereas a rank of 1024 behaved similarly to full fine-tuning, memorizing nearly 50% of the duplicated records.
  • Synergistic Defenses: Combining LoRA with Goldfish loss (which randomly excludes tokens from the loss calculation) or aggressive gradient clipping pushed memorization rates even lower, providing a robust defense-in-depth framework.

Limitations and Future Work

  • Scale of FL Setup: The study focuses on a cross-silo setting with a small number of clients (3). Whether these exact dynamics hold true in massive cross-device federated settings (with millions of clients holding tiny fractions of data) remains an open question.
  • Theoretical Gaps: While the empirical evidence is strong, a rigorous mathematical foundation explaining why low-rank updates specifically target memorization while preserving generalization is still developing.
  • Not a Perfect Silver Bullet: LoRA drastically reduces, but does not completely eliminate, memorization. The authors caution that for absolute privacy guarantees, sensitive data must still be scrubbed before training, and LoRA should be viewed as a powerful mitigation layer rather than a total cure.

Summary updated

Abstract

Federated learning (FL) is a popular paradigm for collaborative training which avoids direct data exposure between clients. However, data privacy issues still remain: FL-trained large language models are capable of memorizing and completing phrases and sentences contained in training data when given their prefixes. Thus, it is possible for adversarial and honest- but-curious clients to recover training data of other participants simply through targeted prompting. In this work, we demonstrate that a popular and simple fine-tuning strategy, low-rank adaptation (LoRA), reduces memorization during FL by a factor of up to 10 without significant performance cost. We study this effect by performing fine-tuning tasks in high-risk domains such as medicine, law, and finance. We observe a reduction in memorization for a wide variety of model families, from 1B to 70B parameters. We find that LoRA can reduce memorization in centralized learning as well, and we compare how the memorization patterns differ. Furthermore, we study the effect of hyperparameters and show that LoRA can be combined with other privacy-preserving techniques such as gradient clipping and Gaussian noise, secure aggregation, and Goldfish loss to further improve record-level privacy while maintaining performance.
BibTeX
@article{bossy2025mitigating,
  author = {Thierry Bossy and Julien Vignoud and Tahseen Rabbani and Juan R. Troncoso Pastoriza and Martin Jaggi},
  title = {{Mitigating Unintended Memorization with LoRA in Federated Learning for LLMs}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=WKPyZnLIW4},
}
Figure