Privacy Scaling Laws for Large Foundation Models

16 concepts13 papers1 proposal

In standard Differential Privacy (DP), adding parameters to a machine learning model is traditionally viewed as a curse. Algorithms like DP-SGD protect individual training samples by injecting noise directly into the model's gradients. Because this noise scales with the dimensionality of the update, conventional wisdom dictated that larger models would suffer catastrophic utility degradation. As Xuechen Li et al. noted in Simple Baselines Are Strong Performers for Differentially Private Natural Language Processing, this noise scaling historically made large-scale models seem fundamentally incompatible with strict privacy budgets.

Yet, a wave of empirical evidence has completely inverted this intuition. Across language and vision domains, scaling up foundation models actually improves their performance under differential privacy constraints, frequently narrowing the gap with non-private baselines. Xuechen Li et al. demonstrated in Large Language Models Can Be Strong Differentially Private Learners that the utility growth of larger pre-trained models outpaces the noise penalty of their higher dimensionality. This phenomenon spans modalities: Zhiqi Bu et al. observed a similar trend in vision transformers (Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy), and Haoran Li 0003 et al. systematically benchmarked this favorable trade-off in PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Models.

Why does scaling up act as an antidote to privacy noise? The answer lies in three core mechanisms: intrinsic dimensionality, pre-trained feature buffering, and high-dimensional geometric dividends.

Mechanism 1: Low Intrinsic Dimensionality

The primary reason large models tolerate DP noise is that they do not utilize their entire parameter space to learn a downstream task. Instead, they operate on a much smaller, restricted manifold known as the task's intrinsic dimensionality.

As Da Yu et al. explain in Differentially Private Fine-tuning of Language Models, while the total capacity of a model grows exponentially, the number of parameters required to solve a specific task remains relatively low. When fine-tuning a high-capacity model using parameter-efficient methods like LoRA, the optimization occurs within this low-dimensional manifold. Because the underlying model is highly capable, the noise injected into the full parameter space fails to disrupt this task manifold. The model's massive capacity effectively dilutes the impact of stochastic perturbations, allowing it to maintain high utility.

Mechanism 2: Pre-trained Knowledge as a Buffer

A model trained from scratch must extract every piece of structural information from noisy gradients. In contrast, a massive pre-trained foundation model already possesses a rich representation of the world.

The authors of Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe argue that pre-existing knowledge acts as a buffer against private fine-tuning noise. Instead of rewriting its internal representations, a large model merely steers them. This effect is explicitly leveraged in activation editing: Anmol Goel et al. showed in Differentially Private Steering for Large Language Model Alignment that larger architectures are less sensitive to noise because they rely heavily on robust internal features acquired during pre-training.

Similarly, in computer vision, Yaodong Yu et al. demonstrated in ViP: A Differentially Private Foundation Model for Computer Vision that high-capacity models initialized with basic non-private features absorb gradient noise far better, stabilizing the entire private training trajectory.

Mechanism 3: The High-Dimensional Privacy Dividend

Geometry changes fundamentally in high dimensions. Rather than acting as a burden, massive dimensionality provides structural advantages for both optimization and representation obfuscation.

Consider optimization constraints: DP-SGD requires per-sample gradient clipping to bound individual influence, which is notoriously memory-intensive. To save memory, practitioners often use group-wise clipping. Zhiqi Bu et al. uncovered a "large model blessing" in On the efficacy of group-wise clipping in differentially private optimization: the accuracy penalty typically introduced by memory-saving group-wise clipping completely vanishes as model capacity scales up.

Furthermore, during inference, high-dimensional spaces provide more "room" to protect data. Zeyu Ma et al. demonstrated this in OSNIP: Breaking the Privacy-Utility-Efficiency Trilemma in LLM Inference via Obfuscated Semantic Null Space. Larger models possess higher-dimensional embedding spaces, creating an abundance of orthogonal directions (a "semantic null space") where user inputs can be obfuscated from servers without degrading the model's ability to process the underlying semantics.

Downstream Implications for Alignment and Generation

This scaling law fundamentally changes how we deploy private AI. Rather than settling for weak, small models to avoid DP noise, practitioners can confidently use massive models for complex downstream tasks.

For instance, in reinforcement learning from human feedback (RLHF), Fan Wu et al. showed in Privately Aligning Language Models with Reinforcement Learning that the performance gap between private and non-private alignment narrows significantly as model capacity increases.

When generating synthetic data or prompt learning, the same rule applies. Franziska Boenisch et al. (Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models) and Da Yu et al. (Privacy-Preserving Instructions for Aligning Large Language Models) both found that more capable, larger models achieve higher task accuracy under strict, low privacy budgets. As summarized by Michele Miranda et al. in Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions, parameter capacity acts as a natural shield, allowing models to absorb stochastic perturbations while maintaining high accuracy.

Key Foundational Work

Large Language Models Can Be Strong Differentially Private Learners (2022) — contributed 1 concept elements to this direction.
Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy (2022) — contributed 1 concept elements to this direction.
Simple Baselines Are Strong Performers for Differentially Private Natural Language Processing (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_privacy_utility_scaling_privacy_utility__bcffbff5,
  title = {privacy-utility scaling (privacy-utility scaling laws)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/privacy-utility-scaling-privacy-utility-scaling-laws/rdr_bcffbff5f45a4cf595f18dd6addfa1ca},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_privacy_utility_scaling_privacy_utility__bcffbff5_v1,
  title = {privacy-utility scaling (privacy-utility scaling laws)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/privacy-utility-scaling-privacy-utility-scaling-laws/rdr_bcffbff5f45a4cf595f18dd6addfa1ca/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake