Introduction
In the pursuit of trustworthy machine learning, researchers have traditionally operated within two distinct silos: Differential Privacy (DP) and Adversarial Robustness. The former focuses on privacy, ensuring that a model's training process does not leak sensitive information about individual data points. The latter focuses on security, ensuring that a model's predictions remain stable even when inputs are subjected to calculated, malicious perturbations.
Despite their shared goal of creating reliable models, these fields have historically utilized different mathematical languages and metrics. This paper provides a theoretical bridge between these two pillars, demonstrating that they are fundamentally two sides of the same mathematical coin. By utilizing the framework of Renyi divergence and randomized algorithms, the authors establish a formal equivalence that allows for the exchange of tools and techniques between the privacy and security domains.
The Problem: Fragmented Trustworthiness
Historically, the definitions of privacy and robustness have targeted different stages of the machine learning pipeline:
- Differential Privacy relies on the principle that a single change in a training dataset should not significantly alter the output distribution of the resulting model.
- Adversarial Robustness relies on the principle that a small change in a specific input at inference time should not change the model's final prediction.
The authors address this fragmentation by asking a fundamental question: Are these actually the same mathematical problem in disguise? By abstracting these concepts, they aim to create a unified framework for "trustworthy AI."
Methodology: Building the Bridge
The authors construct their unified view using two primary theoretical building blocks:
- Probabilistic Mappings (Randomized Algorithms): Rather than analyzing deterministic models where an input $x$ always yields a fixed output $y$, the authors treat models as randomized processes. In this view, for any given input, the model produces a probability distribution over possible outputs.
- Renyi Divergence ($D_\lambda$): This serves as a generalized metric to measure the "distance" between two probability distributions. Renyi divergence is particularly powerful because it encompasses other common measures, such as Kullback-Leibler (KL) divergence and max-divergence, providing a flexible tool for theoretical analysis.
By framing both privacy and robustness in terms of these probabilistic mappings and divergences, the authors derive a Generalized Adversarial Robustness definition. This allows for a direct comparison between a model's sensitivity to input noise (robustness) and its sensitivity to training data changes (privacy).
Key Insights and Theoretical Alignment
The most significant contribution of this work is the formal claim that Renyi-Differential Privacy is equivalent to $D_\lambda$-robustness. Specifically, the paper proves that an algorithm is robust to adversarial perturbations if and only if it satisfies a specific version of Renyi-DP.
Noise as a Universal Tool
A critical insight highlighted by the authors is the role of "noise injection." In the privacy community, adding noise is a standard technique used to mask the presence of individual data points. In the security community, similar randomization techniques—often called "randomized smoothing"—are used to mitigate the impact of adversarial perturbations. The paper demonstrates that these techniques are mathematically identical, serving the same purpose in both domains.
Cross-Pollination of Research
Because the underlying mathematics are equivalent, the authors argue that the mature body of literature surrounding Differential Privacy can be immediately applied to solve adversarial robustness problems. This "cross-pollination" suggests that privacy-preserving training techniques can inherently provide robustness guarantees, and vice versa.
Results and Future Directions
As a theoretical "short note," the paper's results are conceptual rather than empirical:
- Definition 7 (Generalized Adversarial Robustness): The authors successfully formulated a new definition of robustness that applies to randomized classifiers across any metric or divergence.
- Theoretical Convergence: They proved that when parameters are aligned—specifically in cases where the probability of a prediction change is zero—the definitions of a Renyi-private algorithm and a robust algorithm become identical.
Limitations
While the mathematical alignment is robust, the authors note several limitations:
- Empirical Gap: The paper does not provide experimental data on how these equivalences behave under the computational constraints of large-scale neural networks (e.g., ImageNet or MNIST).
- Semantic Differences: The authors acknowledge that while the math aligns, the "semantics" may differ. For instance, "privacy" in an image classification task may not always have a clear real-world interpretation, even if the mathematical definition of DP is satisfied.
Conclusion
By providing a unified view of privacy and robustness, this work lays the groundwork for a more integrated approach to trustworthy machine learning. The authors suggest that future work should focus on using adversarial "attack" frameworks to evaluate the actual privacy guarantees of models, a direction that remains largely unexplored but is now theoretically justified by this unified framework.
Summary updated