Finding ε and δ of Traditional Disclosure Control Systems

Saswat Das, Keyu Zhu, Christine Task, Pascal Van Hentenryck, Ferdinando FiorettoAAAI 2024 · DOI

Introduction

For decades, statistical agencies like the U.S. Census Bureau have relied on intuitive methods to protect data privacy, collectively known as Statistical Disclosure Control (SDC). However, as data science moves toward the rigorous mathematical standard of Differential Privacy (DP), a critical question arises: exactly how much privacy do these traditional methods provide? This paper, presented at AAAI-24, provides a formal bridge between these two worlds, debunking myths about the safety and accuracy of legacy systems.

The Problem: Intuition vs. Rigor

Traditional SDC methods, such as cell suppression and record swapping, lack a formal mathematical definition of privacy. They rely on heuristics—for example, if a table cell contains only one individual, it is suppressed (hidden) to prevent identification. Record swapping involves exchanging attributes between similar individuals to obscure unique identities.

While these methods are intuitive, they do not offer the provable $(\epsilon, \delta)$ guarantees of Differential Privacy. Critics often argue that DP is "too noisy," but without a common framework, it has been impossible to compare the utility and privacy of SDC against DP. By creating randomized "DP-analogs" of traditional SDC, researchers can finally calculate the true privacy loss inherent in legacy systems.

Methods: Mapping SDC to Differential Privacy

The researchers developed a framework to analyze SDC through the lens of $(\epsilon, \delta)$-DP. Because traditional SDC is deterministic (which technically implies infinite privacy loss in a DP context), the authors introduced "carefully randomized" versions:

  1. DP-Cell Suppression ($M_{CS}$): Instead of simply hiding counts below a threshold $k$, this mechanism adds Laplace noise and suppresses the cell only if the noisy count falls below the threshold.
  2. DP-Swapping ($M_{SW}$): This uses a "permute-and-flip" mechanism to select donor records for swapping based on discrepancy scores, allowing for a formal derivation of privacy bounds.
Visualizing traditional SDC methods: (a) raw dataset, (b) histogram, (c) cell suppression, and (d) record swapping.

Key Results

The Privacy Illusion

The study reveals that traditional SDC often provides a false sense of security. While it avoids adding noise to every cell, the probability of total privacy failure ($\delta$) often approaches 1.0. In the context of Differential Privacy, a $\delta$ this high means the mechanism provides almost no meaningful privacy guarantee for many individuals in the dataset.

The Fairness Trap

One of the most significant findings is that SDC is inherently unfair. Because suppression and swapping specifically target outliers and unique records—who are more likely to be members of minority demographic groups—the errors and data distortions introduced by SDC are concentrated on these populations.

Fairness violation (α) increases significantly with swapping probability in traditional SDC methods, highlighting the disproportionate impact on minority groups.

The "fairness violation" ($\alpha$) for SDC swapping was found to be orders of magnitude higher than for the standard Laplace mechanism. For example, at $\epsilon=1$, SDC swapping had a fairness violation of ~1945, while the Laplace mechanism was only ~1.84.

Utility and Accuracy

Contrary to practitioner belief, the simple Laplace mechanism (adding noise to all cells) often produces more accurate results than complex SDC methods for the same level of privacy. In classification tasks using 2019 American Community Survey (ACS) data, models trained on DP-protected data consistently achieved higher ROC-AUC scores than those using data protected by traditional suppression or swapping.

Conclusion

The research concludes that traditional SDC methods are not only mathematically opaque but also frequently less secure, less accurate, and significantly less fair than modern Differential Privacy. The authors urge data agencies to move away from intuitive but flawed legacy methods in favor of the rigorous, equitable framework provided by DP.

Summary updated

BibTeX
@inproceedings{das2024finding,
  author = {Saswat Das and Keyu Zhu and Christine Task and Pascal Van Hentenryck and Ferdinando Fioretto},
  title = {{Finding ε and δ of Traditional Disclosure Control Systems}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i20.30204},
}
Figure