Machine learning models leak more than individual records. A model trained on a hospital's data can reveal what fraction of its patients were women, or how common a disease was. This is called property inference, and the standard privacy guarantee, differential privacy, does not stop it. Differential privacy hides any one person's influence. A population-wide proportion is the sum of thousands of people, so the per-person noise averages out and the proportion survives (Hacking smart machines with smarter ones). Separately, a well-known result says that training a model to resist small worst-case input perturbations (adversarial training) costs it accuracy on ordinary inputs. The explanation is that the model must give up "non-robust features": many weak, individually fragile statistical cues that are genuinely predictive in aggregate (Robustness May Be at Odds with Accuracy).
This note asks whether those two facts are one fact. Non-robust features are fine-grained correlations whose strength depends on the exact makeup of the training set. If they are also where a model encodes its training population's proportions, then the clean accuracy adversarial training gives up is partly purchasing protection of those proportions, and the model never needs to be told which property is sensitive. The rival prediction is just as plausible. Adversarially trained models memorize the neighborhoods of their training points and leak membership 3.5x more on CIFAR-10 (Privacy Risks of Securing Machine Learning Models against Adversarial Examples), which suggests they could leak proportions more too. We propose to settle which prediction holds, and to measure it in a way that is not fooled by the trivial fact that any weaker model leaks less.
Core Claim
The testable claim is this: at equal clean accuracy, adversarially trained models leak training-set proportions less than models degraded by other, property-agnostic means. The mechanism should be visible directly. In standard models, the signal that reveals the training proportion should be removable by tiny input perturbations. In robust models, whatever proportion signal remains should survive large perturbations.
If this holds, the robustness–accuracy trade-off has a measurable privacy component that nobody has priced. If it fails, and robust models leak more at matched accuracy, then the memorization story from Song et al. extends from individuals to populations. That would be a warning for anyone deploying robust models on sensitive cohorts. Either outcome is publishable, and the experiments are built so that both are decisive.
The quantity we care about is the leakage gap at matched accuracy. Write $L(m)$ for the best property-inference attack's success at telling whether model $m$ was trained on population mix $G_0$ or $G_1$. Write $a(m)$ for its clean test accuracy. The privacy dividend of an adversarially trained model $m_{\text{adv}}$ is
$$D = L(m_{\text{ctrl}}) - L(m_{\text{adv}}) \quad \text{where } a(m_{\text{ctrl}}) = a(m_{\text{adv}}),$$
with $m_{\text{ctrl}}$ a control model degraded to the same clean accuracy without adversarial training. A positive $D$ means the robustness tax buys privacy beyond what lost accuracy alone explains.
Relation to Prior Work
Three papers motivate the question without answering it.
Song et al. evaluate six adversarial defenses against membership inference only. They never ask whether a robust model reveals its training population's composition (Privacy Risks of Securing Machine Learning Models against Adversarial Examples). Tsipras et al. supply the robust/non-robust feature picture that generates our hypothesis, but they do not study privacy (Robustness May Be at Odds with Accuracy). Like Oil and Water shows that two robustness goals can collide on minority subpopulations: group robustness against spurious correlations, and resistance to training-data poisoning. That collision constrains us, because any defense that reshapes how minority groups are fit may change proportion leakage. But that paper does not involve privacy attacks.
The attack methods come from three sources. Formalizing Distribution Inference Risks frames property inference as a game in which a trainer secretly picks $G_0$ or $G_1$. It supplies three attacks: a loss test, a threshold test, and meta-classifiers, which are networks that read a target model's weights directly. SNAP supplies the strongest active attack. It inserts a small number of label-flipped training examples into a subpopulation, which amplifies the proportion signal, and it works even when only hard labels are visible. SNAP matters for us because adversarial training does nothing explicit against label flipping, so robustness could suppress passive leakage while leaving amplified leakage intact.
Two papers supply the property-aware defenses we compare against. Inf2Guard minimizes mutual information between internal representations and a known sensitive variable. It never tests adversarial robustness, and it requires the sensitive property to be named in advance. Protecting Global Properties of Datasets with Distribution Privacy Mechanisms adds noise along the known direction of property variation. It works for releasing tabular statistics, and the authors leave deep classifiers as open work.
Our proposal differs from all of these in one respect. We study a defense that is blind to the property and was built for a different threat. The question is whether its known accuracy cost is partly a hidden privacy benefit.
Research Program
We train families of models on CelebA, RSNA Bone Age, and CIFAR-10. In each family, the trainer varies one sensitive proportion between $G_0$ and $G_1$ across a range of gaps:
- CelebA: the fraction of male faces.
- RSNA Bone Age: the fraction of female patients.
- CIFAR-10: the ratio between two semantically close classes.
CelebA and RSNA Bone Age were chosen because prior work already shows strong leakage on them (Formalizing Distribution Inference Risks, SNAP). CIFAR-10 is the standard setting for adversarial training. The cat-versus-dog style pairs are where Robustness May Be at Odds with Fairness shows adversarial training hurts clean accuracy most, so a CIFAR-10 class-ratio property tests whether the privacy dividend concentrates where the accuracy tax does.
For each dataset we sweep the adversarial training budget $\epsilon$ in $\ell_\infty$ and $\ell_2$, from zero upward. This traces clean accuracy against leakage under every attack. We then build accuracy-matched controls at each point on the curve and compute $D$.
The mechanism analysis is the part that makes this more than a measurement. For each trained model we take a fixed set of probe inputs and compute a proportion signal: the difference in the model's outputs on those probes between versions trained on $G_0$ and on $G_1$. We then ask how large an input perturbation must be to erase that signal. Concretely, we search for perturbations that minimize the signal on each probe, and we record the radius at which attack success falls to chance. We call this the robustness radius of the leakage. Our hypothesis predicts a small radius for standard models and a large one for robust models, and it predicts that the radius tracks the adversarial training budget.
We add a second view of the mechanism through representations. We fit linear probes for the sensitive attribute and for the proportion signal on the internal representations of standard and robust models. Then we test whether the proportion signal in standard models lives mostly in directions that adversarial training removes. This also answers the entanglement concern raised for latent-space defenses by PriSampler: do robust representations separate the property from task features more cleanly?
Baselines
Standard training. The naive reference point is standard training with no defense. It fixes the leakage ceiling and the accuracy ceiling for every dataset.
Gaussian augmentation. The ablation that isolates our key ingredient is training with random Gaussian input noise. We increase the noise until clean accuracy matches each adversarially trained model. This control applies input perturbation of the same magnitude family but not worst-case perturbation. If adversarial training beats it at matched accuracy, the dividend comes from removing non-robust features specifically, not from smoothing in general.
Capacity and early-stopping controls. A second accuracy-matched control reaches the same clean accuracy by early stopping or by using a smaller network. This rules out the explanation that any less-fit model leaks less.
DP-SGD. Differentially private stochastic gradient descent (DP-SGD) is the record-level differential privacy baseline, tuned to the same clean accuracy points. Prior work suggests it should do poorly here. On CIFAR-10 it needed to fall to 48% utility to match the 51% membership-attack accuracy that Inf2Guard reached at 77% utility. Its leakage against property attacks at matched accuracy is the comparison that speaks most directly to the limits of differential privacy.
Inf2Guard. The strongest existing defense is Inf2Guard, given the sensitive property it needs. It reports cutting Census property-inference accuracy from 68% to 34% "without sacrificing task accuracy" (Inf2Guard). We run it on our image tasks with the same architecture, training data, and hyperparameter search budget as adversarial training. We also reproduce its Census number first, as a sanity check. For the distribution-privacy noise mechanisms, Chen and Ohrimenko report attack accuracy falling from 75% to near-random 50% on Adult, but for released statistics rather than trained classifiers. We use that figure only as a reference for what "near chance" looks like.
What counts as success. I found no reported result on Lacuna for property inference against adversarially trained models. Our curves would therefore be the first. Beating the controls means adversarial training shows lower leakage than Gaussian augmentation, early stopping, and DP-SGD at every matched accuracy, with confidence intervals across at least five seeds. We do not expect adversarial training to beat Inf2Guard outright, because Inf2Guard knows the target property. Matching it within a few points while being blind to the property would be a strong result.
Why not just…?
Why not just use DP-SGD? Record-level noise is calibrated to one person, and the proportions we target are aggregates that survive that noise (Hacking smart machines with smarter ones). SNAP reports that differential privacy only partially mitigates its attack (SNAP). DP-SGD is still in our baseline set at matched accuracy, so this objection is tested directly rather than argued away.
Why not just use Inf2Guard or directional noise, since they target the property directly? When the sensitive property is known in advance, they probably should win, and we expect them to. Their limitation is the requirement itself. They need the property named, and the directional mechanism also needs the direction of variation, which is rarely available in deep models (Protecting Global Properties of Datasets with Distribution Privacy Mechanisms). In practice an attacker may target a property the defender never anticipated. We therefore run a held-out-property test. Inf2Guard is trained to hide property A and then attacked on property B, while adversarial training is attacked on both. If Inf2Guard protects unnamed properties equally well, that weakens our practical pitch, and we will report it as such.
Why not just rebalance the training data on the sensitive attribute? If the attribute is labeled, rebalancing hides that one proportion trivially, at the cost of discarding data. It does not hide correlated properties. It also requires labels that Like Oil and Water notes are usually missing. In addition, Robustness May Be at Odds with Fairness shows that balanced data does not remove adversarial training's uneven accuracy cost. We include rebalancing as a property-aware reference in the held-out-property test.
Why not just reveal only hard labels or rescale output confidences? SNAP remains effective when only hard labels are returned (SNAP). Adjusting the output temperature is easily bypassed once the attacker accounts for it (Can We Infer Confidential Properties of Training Data from LLMs?). Meta-classifiers also read model weights directly, which output-level fixes cannot touch.
Decisive Experiments
Matched-accuracy comparison. On CelebA, RSNA Bone Age, and CIFAR-10, we compare $D$ against all three property-agnostic controls, using the loss test, threshold test, and meta-classifier attacks. A consistently positive $D$ that grows with $\epsilon$ supports the dividend hypothesis. A consistently negative $D$ supports leakage amplification through memorization.
SNAP poisoning test. At poisoning rates of 0.4% to 1.5%, the budget in SNAP, which reports 91% distinguishing success against undefended models, we attack robust and control models in both logit and label-only settings. If robustness suppresses passive leakage but not SNAP, the dividend is real but shallow. That result would directly motivate pairing robustness with poisoning defenses, while respecting the minority-group damage those defenses cause (worst-group accuracy dropping from 55% to 41% under EPIc on CelebA, per Like Oil and Water).
Mechanism check. The hypothesis fails as stated if the leakage robustness radius does not differ between standard and robust models. In that case any matched-accuracy difference has some other cause, and we will say so rather than claim the non-robust-feature explanation.
Concentration check. On CIFAR-10 we test whether the dividend concentrates on the class pairs where adversarial training costs the most clean accuracy. A positive result would tie the privacy gain to the specific accuracy that robustness gives up, not merely to robustness in general.
References
- Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers
- Robustness May Be at Odds with Accuracy
- Privacy Risks of Securing Machine Learning Models against Adversarial Examples
- Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Odds
- Formalizing Distribution Inference Risks
- SNAP: Efficient Extraction of Private Properties with Poisoning
- Inf2Guard: An Information-Theoretic Framework for Learning Privacy-Preserving Representations against Inference Attacks
- Protecting Global Properties of Datasets with Distribution Privacy Mechanisms
- Robustness May Be at Odds with Fairness: An Empirical Study on Class-wise Accuracy
- PriSampler: Mitigating Property Inference of Diffusion Models
- Can We Infer Confidential Properties of Training Data from LLMs?