Single Run Auditing for Differentially Private Models

This direction is merged into another direction: shadow model computational overhead. Its last version stays readable and citable here.

Version 1 of 118 papers20 conceptsfrozen 2026-10-04

Differential Privacy (DP) offers a rigorous mathematical promise: the output of an algorithm should not depend too heavily on any single individual's data. In machine learning, this is typically achieved via DP-SGD, where gradients are clipped and noise is added during training. However, a persistent "privacy gap" exists between the theoretical guarantees written in a paper and the actual privacy leaked by a specific implementation.

To bridge this gap, researchers use privacy auditing—an empirical attempt to "break" the model's privacy to see how much information actually leaks. Until recently, this was a computationally Herculean task, often requiring thousands of independent training runs. A new wave of research is transforming this field by enabling tight, rigorous privacy audits using only a single training run.

The Computational Bottleneck of Privacy Auditing

The standard way to audit a DP model is to treat it as a game of Membership Inference. An auditor asks: "Can I tell if this specific person was in the training set?" To answer this with statistical confidence, you traditionally need to train two versions of a model: one with the person (the "in" model) and one without (the "out" model).

Because machine learning is stochastic, you can't just do this once. You need to train hundreds or thousands of "in" and "out" model pairs to build a distribution of outcomes. As noted by Thomas Steinke and Matthew Jagielski in Privacy Auditing with One (1) Training Run, this creates a massive hurdle for modern large-scale models. If training a single LLM costs millions of dollars, training it 2,000 times to verify its privacy is impossible.

The Intuition of Canaries

The breakthrough in efficient auditing relies on the concept of canaries. Instead of retraining the whole model to test one data point, researchers insert multiple "canary" points into a single training run.

If these canaries are designed correctly—for example, by using uncorrelated synthetic data—they act as independent probes. Ruixuan Liu and Li Xiong demonstrate in UniAud that by using a multi-task learning approach, one can decouple the auditing of these canaries from the actual utility of the model. This allows for $O(1)$ complexity: you get a high-confidence privacy bound in the same time it takes to train the model once.

Beyond Membership: What Are We Actually Protecting?

Auditing isn't just about checking if the code is correct; it's about checking if we are measuring the right thing. Most DP libraries use "add/remove" adjacency, which protects the fact of a person's participation. However, Gauri Pradhan and Antti Honkela argue in Beyond Membership: Limitations of Add/Remove Adjacency in Differential Privacy that this often overstates protection.

If an attacker already knows you are in a medical dataset but wants to find out your specific diagnosis (a "substitute" adjacency problem), the privacy leakage can be significantly higher than standard tools report. By developing novel auditing canaries that target specific attributes, these researchers have shown that real-world leakage often matches much stricter privacy metrics than practitioners expect.

Cryptographic vs. Empirical Verification

While empirical auditing uses canaries to "test" the model, another direction seeks to "prove" the training process itself. Ali Shahin Shamsabadi and Nicolas Papernot proposed Confidential-DPproof, which uses zero-knowledge proofs (ZKP). Instead of running membership attacks, the trainer provides a cryptographic certificate that the gradients were indeed clipped and noised according to the DP-SGD protocol.

This complements empirical work: while ZKPs prove the algorithm was followed, empirical audits by researchers like Zihang Xiang in Tight Privacy Audit in One Run ensure that the implementation (like how data is shuffled) doesn't introduce unexpected leaks. For instance, Meenatchi Sundaram Muthu Selva Annamalai found that replacing Poisson sampling with simple shuffling can lead to 4x more privacy leakage than theoretical bounds suggest, a gap that only rigorous, efficient auditing can catch.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_single_run_auditing_for_differentially_p_8a1a5078,
  title = {Single Run Auditing for Differentially Private Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-run-auditing-for-differentially-private-models/rdr_8a1a5078c6434dffb34d644419425a3c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_single_run_auditing_for_differentially_p_8a1a5078_v1,
  title = {Single Run Auditing for Differentially Private Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-run-auditing-for-differentially-private-models/rdr_8a1a5078c6434dffb34d644419425a3c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}