The Gap Between Privacy Theory and Empirical Auditing

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2350 papers465 conceptsfrozen 2026-10-04

In the field of privacy-preserving machine learning, there is a growing realization that the mathematical guarantees we use to protect data do not always align with the actual risks models face. This discrepancy is often referred to as the "privacy gap"—a "no-man's-land" between the pessimistic upper bounds of theoretical privacy and the optimistic lower bounds of empirical attacks.

When a researcher trains a model with Differential Privacy (DP), they calculate a privacy budget, $\epsilon$ (epsilon). A low $\epsilon$ suggests strong protection, while a high $\epsilon$ suggests vulnerability. However, as Daniel Bernau and colleagues observed in Assessing differentially private deep learning with Membership Inference, these theoretical parameters frequently fail to reflect a model's actual susceptibility to Membership Inference Attacks (MIAs).

The Pessimistic Upper Bound

Theoretical differential privacy is designed to be a "worst-case" guarantee. It assumes an omniscient adversary with full knowledge of the model and the ability to manipulate data in the most damaging ways possible. Because of this, the mathematical proofs often produce privacy loss estimates that are orders of magnitude higher than what is observable in practice.

Ananth Raghunathan, Ilya Mironov, and their co-authors argue in That which we call private that while "relaxed" definitions like Rényi DP (RDP) help provide tighter bounds, a substantial gap remains. This looseness creates a usability barrier: practitioners may reject a model as "unsafe" because its theoretical $\epsilon$ is high, even if it is practically immune to data extraction. For instance, Milad Nasr and Thomas Steinke show in The Last Iterate Advantage that standard analysis is overly pessimistic because it assumes an attacker sees every intermediate update during training, rather than just the final model.

The Optimistic Lower Bound

On the other side of the gap are empirical audits—attacks designed to "test" the privacy of a model. If an attack fails to recover training data, we might assume the model is safe. However, this is an "optimistic" view because the failure of an attack might stem from a weak adversary rather than a strong defense.

Yuhan Liu and Florent Guépin demonstrate in Sub-optimal Learning in Meta-Classifier Attacks that standard auditing classifiers often get stuck on simple rules, leading to a dangerous underestimation of risk. Similarly, Ashwinee Panda and Milad Nasr point out in Privacy Auditing of Large Language Models that if the "canaries" (test sequences) used for auditing are too easy for the model to ignore, the audit will produce a loose lower bound, providing a false sense of security.

Why Epsilon is an Incomplete Proxy

The most significant finding across this research is that $\epsilon$ alone is an insufficient proxy for reconstruction risk. Jamie Hayes and Borja Balle show in Bounding training data reconstruction in DP-SGD that two models with the exact same $\epsilon$ can have vastly different levels of data leakage depending on hyperparameters like sampling rates or step counts.

This is further complicated by "aggressive" relaxations of privacy. Zeyu Ding reveals in Reconstruction Attacks on Aggressive Relaxations of Differential Privacy that some frameworks can claim a near-zero privacy loss while still allowing an attacker to reconstruct an entire dataset with near-perfect accuracy. This happens when the noise addition process is not truly independent of the data distribution, rendering the formal guarantee meaningless.

Closing the Gap

To bridge this divide, researchers are moving in two directions:

  1. Tighter Accounting: Developing mathematical frameworks that better reflect real-world training. Saeed Mahloujifar and colleagues in Optimal Membership Inference Bounds provide tighter bounds by directly analyzing the Gaussian mechanism rather than converting to standard $(\epsilon, \delta)$ units.
  2. Stronger Auditing: Creating "worst-case" empirical attacks. Tudor Cebere and Nicolas Papernot introduced "gradient-crafting" in Tighter Privacy Auditing of DP-SGD, which uses adversarial gradients to find the true limits of privacy leakage, effectively closing the gap between theory and practice.

The goal of this research direction is to move toward a "Rosetta Stone" for privacy—a way to translate empirical observations of memorization into rigorous lower bounds for privacy parameters, as explored by Matthew Jagielski in A Note On Interpreting Canary Exposure. Only when our theoretical upper bounds and empirical lower bounds converge can we truly understand the privacy of the models we deploy.

Papers in this version

Showing 60 of 350 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 405 papers
  2. Version 1 · 2026-10-04 · 350 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_theoretical_empirical_privacy_gap_df8fb4cb,
  title = {theoretical-empirical privacy gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/theoretical-empirical-privacy-gap/rdr_df8fb4cb55e84188a52d044e64848123},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_gap_between_privacy_theory_and_empir_df8fb4cb_v1,
  title = {The Gap Between Privacy Theory and Empirical Auditing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-gap-between-privacy-theory-and-empirical-auditing/rdr_df8fb4cb55e84188a52d044e64848123/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}