Private Fine-tuning of Large Language Models with Zeroth-order Optimization

Xinyu Tang, Ashwinee Panda, Milad Nasr, Saeed Mahloujifar, Prateek Mittal

TL;DR

The paper introduces DP-ZO, a highly memory-efficient framework for differentially private fine-tuning of large language models (LLMs). By leveraging zeroth-order optimization, DP-ZO avoids high-dimensional gradient privatization and instead adds noise to a single scalar value (the loss difference). This allows it to scale seamlessly to models with up to 66 billion parameters, bypassing the massive memory bottlenecks of traditional DP-SGD.

What Problem Does This Paper Address?

Training or fine-tuning LLMs on private data requires strict privacy guarantees to prevent the model from memorizing and leaking sensitive information. The gold standard for this is Differential Privacy (DP), typically implemented via Differentially Private Stochastic Gradient Descent (DP-SGD).

However, DP-SGD has a massive scalability problem: it requires computing and clipping gradients per individual training example. For large models, this per-example gradient clipping explodes memory consumption, slows down training, and demands complex engineering to distribute across multiple GPUs. As a result, scaling DP-SGD to foundation models beyond a few billion parameters is incredibly challenging.

Methods

The authors propose DP-ZO, which combines differential privacy with zeroth-order (ZO) optimization—specifically, Simultaneous Perturbation Stochastic Approximation (SPSA).

Instead of computing exact gradients via backpropagation, zeroth-order optimization estimates the gradient using only forward passes:

  1. A random perturbation vector ($z$) is generated.
  2. The model's loss is evaluated twice: once with parameters shifted slightly forward ($\theta + \phi z$) and once shifted backward ($\theta - \phi z$).
  3. The difference between these two losses determines the scalar step size along the random direction.

The key insight is that because the direction vector ($z$) is completely random and independent of the training data, the only private information leaked from the dataset is the scalar loss difference.

Figure 1: Visualization of DP-ZO. The only informa- tion from private data is a scalar step size for direction with lower target function value and we only need to add noise to this scalar. This scalar privatization enjoys the benefits of flexibility with DP mechanisms, ease of implementation, and reduced computation.

To make this algorithm differentially private, DP-ZO simply clips this scalar loss difference to bound its sensitivity and adds one-dimensional noise (either Gaussian or Laplace) before updating the weights. This completely eliminates the need for backpropagation and per-example gradient clipping.

Datasets

The framework was evaluated primarily in a challenging few-shot setting (using 1,000 sampled training points), which reflects realistic data-constrained private fine-tuning scenarios. The benchmark NLP tasks included:

  • SQuAD and DROP: Question answering and text generation tasks (evaluated using F1 score).
  • SST2: Text classification for sentiment analysis (evaluated using accuracy).
  • QNLI: A binary classification dataset used to evaluate performance when scaling up the number of training samples.

Key Insights and Ideas

  • Defeating the Curse of Dimensionality: In DP-SGD, noise is added to a high-dimensional gradient vector, which heavily degrades model utility. DP-ZO adds noise to a single dimension (a scalar), meaning the noise is far less destructive to the optimization process.
  • Unlocking Pure $\varepsilon$-DP: The Laplace mechanism provides a stronger, "pure" privacy guarantee ($\delta=0$), but it suffers from high variance in high dimensions, making it useless for DP-SGD. Because DP-ZO operates on a scalar, the Laplace mechanism becomes highly effective, providing the first non-trivial pure $\varepsilon$-DP utility for large models.
  • The Scaling Synergy: Crucially, the authors discovered that as the model size increases, the utility gap between private and non-private models shrinks. Larger models appear more robust to the scalar noise introduced by DP-ZO.

Key Results

  • Seamless Scalability: While DP-SGD ran out of memory (OOM) on a single A100 GPU when fine-tuning models larger than 13B parameters, DP-ZO scaled effortlessly up to OPT-66B.
  • High Utility: On the SQuAD dataset using OPT-66B with LoRA, DP-ZO achieved an F1 score of 84.12% under a conservative privacy budget ($\varepsilon=1$), coming remarkably close to the non-private baseline of 87.49%.
Table 2: DP-ZO (Gaussian) and DP-SGD with full parameter and LoRA fine-tuning on SQuAD with 1000 training samples across different model sizes. (1, 10−5)-DP. ‘−’ means the approach did not scale with straightforward implementation; Section 5 details the additional engineering required to scale DP-SGD to larger models. ‘−−’ for DP-ZO means the results are omitted due to limited computational resources. Due to limited computing resources, this table does not include the standard deviation for OPT-66B model.
  • Pure DP Breakthrough: Under pure $\varepsilon$-DP ($\varepsilon=4$) on SQuAD, DP-ZO achieved a 73.52% F1 score, whereas DP-SGD failed to improve over the zero-shot baseline (~47%).
  • Extreme Memory Efficiency: For a sequence length of 2048 on an OPT-2.7B model, all DP-SGD variants encountered OOM errors, while DP-ZO maintained a memory footprint of under 16GB. Under strict 8GB on-device memory constraints, DP-ZO was the only method capable of training.

Limitations and Future Work

The primary limitation of DP-ZO is its data efficiency. Zeroth-order optimization inherently requires more training iterations to converge compared to first-order methods like SGD because it relies on random direction sampling.

As the dataset size scales up to the full training sets (e.g., 100k+ samples), DP-SGD eventually outperforms DP-ZO in utility because it utilizes exact gradient directions. The authors note that improving the data efficiency of private zeroth-order optimization remains an open and promising direction for future research.

Summary updated

Abstract

Differentially private stochastic gradient descent (DP-SGD) allows models to be trained in a privacy-preserving manner, but has proven difficult to scale to the era of foundation models. We introduce DP-ZO, a private fine-tuning framework for large language models by privatizing zeroth order optimization methods. A key insight into the design of our method is that the direction of the gradient in the zeroth-order optimization we use is random and the only information from training data is the step size, i.e., a scalar. Therefore, we only need to privatize the scalar step size, which is memory-efficient. DP-ZO provides a strong privacy-utility trade-off across different tasks, and model sizes that are comparable to DP-SGD in $(\varepsilon,δ)$-DP. Notably, DP-ZO possesses significant advantages over DP-SGD in memory efficiency, and obtains higher utility in $\varepsilon$-DP when using the Laplace mechanism.
BibTeX
@article{tang2025private,
  author = {Xinyu Tang and Ashwinee Panda and Milad Nasr and Saeed Mahloujifar and Prateek Mittal},
  title = {{Private Fine-tuning of Large Language Models with Zeroth-order Optimization}},
  journal = {Trans. Mach. Learn. Res.},
  year = {2025},
}
Figure