Single-Run Privacy Auditing for Large-Scale Models

This direction is merged into another direction: theoretical-empirical privacy gap. Its last version stays readable and citable here.

Version 1 of 114 papers22 conceptsfrozen 2026-10-04

Differential Privacy (DP) offers a rigorous mathematical guarantee that a machine learning model will not leak sensitive information about individual training examples. However, there is often a gap between the theoretical privacy budget ($\epsilon$) and the actual privacy leakage of a specific implementation. To bridge this gap, researchers use privacy auditing: an empirical process that tests how easily an adversary can detect whether a specific "canary" data point was included in the training set.

Traditionally, auditing is a computational nightmare. To get a statistically significant estimate of privacy leakage, you must train the same model thousands of times—half the time with the canary, and half the time without it—to observe the distribution of the model's outputs. For modern Large Language Models (LLMs) that cost millions of dollars to train once, this "multi-run" approach is impossible.

A new research direction, often called One-Run Auditing (ORA) or One-Shot Estimation, solves this by parallelizing the auditing process within a single training execution.

The Intuition: Parallelizing Canaries

The core insight behind one-run auditing is that instead of running 1,000 training sessions to test one canary, we can run one training session containing 1,000 independent canaries.

In this setup, each canary is injected into the training set with a fixed probability (usually 50%). Because the inclusion of each canary is an independent random variable, the final trained model effectively contains the results of thousands of simultaneous "mini-experiments." As Thomas Steinke, Milad Nasr, and Matthew Jagielski demonstrate in Privacy Auditing with One (1) Training Run, treating these inclusions as independent trials allows researchers to calculate privacy lower bounds with high confidence using only the final model weights or gradients.

Auditing Distributed and Federated Systems

This approach is particularly natural in Federated Learning (FL), where thousands of clients contribute updates. Galen Andrew, Peter Kairouz, and colleagues in One-shot Empirical Privacy Estimation for Federated Learning showed that by inserting "canary clients" with random synthetic updates, one can measure how much the global model "memorizes" these specific vectors.

The intuition here is that different dimensions of the model's parameter space can act as parallel experiments. If the training process is private, the noise added during optimization should mask the influence of any single canary client across these dimensions, making them indistinguishable from clients who were not included.

Tightening the Bounds

While the "one-run" concept drastically reduces cost, early versions often produced "loose" bounds—estimates that significantly underestimated the true privacy leakage. Recent work has focused on making these audits "tighter" (more accurate) through several mechanisms:

  1. $f$-Differential Privacy: Instead of just looking at $(\epsilon, \delta)$, Saeed Mahloujifar, Kamalika Chaudhuri, and Luca Melis utilize the $f$-DP framework. In Auditing f-Differential Privacy in One Run, they use trade-off curves to produce more granular empirical estimates that track theoretical limits more closely than traditional methods.
  2. Advanced Membership Inference: The "attack" used to detect the canary matters. Terrance Liu and Matteo Boglioni introduced Quantile Regression-Based Membership Inference, which improves the detection of canaries in black-box settings, resulting in privacy bounds up to 3x tighter than naive counting.
  3. Bayesian Frameworks: By modeling the joint distribution of attack error rates rather than treating them as isolated events, researchers like Santiago Zanella-Béguelin and Ahmed Salem have shown that we can extract more information from fewer observations, further increasing efficiency (Bayesian Estimation of Differential Privacy).

The Interference Challenge

The main limitation of one-run auditing is interference. If you inject too many canaries into a single run, they begin to interact with each other, potentially masking the signal of individual privacy leaks.

Amit Keinan, Moshe Shenfeld, and Katrina Ligett mathematically formalized this in How Well Can Differential Privacy Be Audited in One Run?. They identified that as the number of canaries increases, the "noise" from other canaries makes it harder to detect any single one, leading to an underestimation of leakage. Solving this requires a careful balance: using enough canaries to get statistical power, but not so many that they interfere with the audit's sensitivity.

By moving from sequential retraining to parallel canary injection, this research direction makes it possible to verify the privacy of the world's largest models, ensuring that "private" AI is a measurable reality rather than just a theoretical promise.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_single_run_privacy_auditing_for_large_sc_b191123c,
  title = {Single-Run Privacy Auditing for Large-Scale Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-run-privacy-auditing-for-large-scale-models/rdr_b191123cdb244032858e602436f18f62},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_single_run_privacy_auditing_for_large_sc_b191123c_v1,
  title = {Single-Run Privacy Auditing for Large-Scale Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-run-privacy-auditing-for-large-scale-models/rdr_b191123cdb244032858e602436f18f62/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}