How reparametrization trick broke differentially-private text representation learning

Ivan HabernalACL 2022 · DOI

Introduction

Differential Privacy (DP) provides a rigorous mathematical framework for privacy-preserving machine learning. However, its implementation in deep learning architectures, such as Variational Autoencoders (VAEs), is fraught with subtle pitfalls. This paper, presented at ACL 2022, highlights a catastrophic failure in DPText, a framework designed for private text representation learning. The failure stems from a misapplication of the reparametrization trick, leading to infinite privacy loss despite claims of low privacy budgets ($\epsilon$).

The Reparametrization Trap

To enable end-to-end training, researchers often use the reparametrization trick to sample noise differentiably. In the case of DPText, the authors intended to sample from a Laplace distribution using the inverse Cumulative Distribution Function (CDF) method.

Extracted figure from page 3 of How reparametrization trick broke differentially-private text representation learning

The core issue lies in the transformation formula. DPText utilized a formula designed for a uniform distribution centered at zero ($[-0.5, 0.5]$) but applied it to a standard uniform distribution ($[0, 1]$). As shown in the figure above, the red function is not monotonic over the $[0, 1]$ interval. For half of the input range ($u > 0.5$), the function produces complex numbers or NaNs, which are typically zeroed out in practical implementations. This creates a non-continuous output space where certain values are impossible for specific inputs, fundamentally violating the requirements for Differential Privacy and resulting in infinite privacy loss.

Empirical Verification and Results

To catch such failures, the author proposes a simple "reconstruction attack" as an empirical sanity check. By testing the mechanism on extreme "neighboring" datasets—for instance, a vector of all zeros versus a vector of all ones—one can estimate the actual privacy loss.

Comparison of estimated actual privacy loss across different mechanisms. DPText shows infinite loss, while ADePT reveals a privacy loss that scales poorly with dimensionality.

The empirical results confirm the theoretical findings. As seen in the multi-panel comparison, the standard Laplace mechanism behaves as expected, with the estimated loss staying below the theoretical bound. However, the DPText mechanism exhibits infinite privacy loss across all desired $\epsilon$ values. Furthermore, the analysis of another system, ADePT, reveals that its privacy guarantees degrade rapidly as the dimensionality of the representation increases, due to incorrect sensitivity calculations.

Conclusion

The failure of DPText serves as a cautionary tale: high utility at very low privacy budgets is often a red flag for implementation bugs rather than a breakthrough. The author advocates for the adoption of empirical sanity checks as a standard requirement for peer-reviewed research in privacy-preserving NLP. While these checks provide a lower bound and cannot replace formal proofs, they are essential for detecting catastrophic implementation errors that lead to "privacy theater."

Summary updated

BibTeX
@inproceedings{habernal2022how,
  author = {Ivan Habernal},
  title = {{How reparametrization trick broke differentially-private text representation learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-short.87},
}
Figure