When we evaluate the privacy of a machine learning model, we typically rely on aggregate statistics. We might measure the average success rate of a membership inference attack (MIA) or compute a global privacy budget ($\epsilon$) using Differential Privacy (DP). However, a growing body of research shows that privacy risk is fundamentally non-uniform.
For the vast majority of individuals in a dataset, privacy is naturally protected because their data points blend into the crowd. But for a small subset of statistical outliers, minority subgroups, or rare attributes, the risk is catastrophic. As Nicholas Carlini et al. demonstrated in Extracting Training Data from Diffusion Models, state-of-the-art generative models do not just learn abstract concepts—they memorize and can be forced to regenerate specific training examples. Crucially, data points that are statistical outliers within a model's embedding space are significantly more vulnerable to this memorization and extraction.
Why Outliers Leak: The Intuition of Model Surprise
To understand why outliers are uniquely vulnerable, we must look at how neural networks learn. During training, a model optimizes its weights to minimize overall loss. When it encounters a typical data point that aligns with the majority distribution, the model can already classify or generate it accurately; the required weight adjustments are minimal.
However, when the model encounters an outlier or a rare attribute, it experiences a high error signal. To accommodate this "surprising" data point, the model must make pronounced adjustments. This learning dynamic leaves distinct, indelible footprints in the model:
- Gradient Magnitudes: In distributed settings like federated learning, unique or difficult samples elicit much larger gradient updates. Hongxu Yin and Jan Kautz showed in See Through Gradients: Image Batch Recovery via GradInversion that these larger gradient magnitudes make unique samples disproportionately easy to reconstruct from shared updates. Similarly, Hongsheng Hu et al. found in Source Inference Attacks in Federated Learning that high data heterogeneity creates distinct signatures in local updates, allowing adversaries to trace specific records back to individual clients.
- Loss Surface Curvature: The geometric landscape of the loss function around a data point directly correlates with its privacy risk. In Unveiling Privacy, Memorization, and Input Curvature Links, the authors proved that high input loss curvature consistently identifies rare or difficult outlier examples that are highly susceptible to memorization.
- Information Density: From an information-theoretic perspective, rare data points carry higher information density. Jiachun Liao et al. established in Hypothesis Testing in the High Privacy Limit that rare symbols are inherently more informative to adversaries, meaning that uniform privacy protections fail to safeguard them adequately.
Because of these mechanisms, certain "sitting duck" data points remain consistently vulnerable across entirely different target models and attack strategies, a phenomenon systematically quantified by Mauro Conti et al. in On the Vulnerability of Data Points Under Multiple Membership Inference Attacks and Target Models.
The Masking Effect of Aggregate Metrics
If privacy risk is concentrated heavily in the tails of a distribution, standard evaluation metrics like average accuracy or Area Under the Curve (AUC) become dangerously deceptive. A membership inference attack might have a low average success rate across an entire dataset, yet achieve near-100% certainty when targeting specific outliers.
To expose this vulnerability, Nicholas Carlini et al. developed the Likelihood Ratio Attack (LiRA) in Membership Inference Attacks From First Principles. By shifting the evaluation metric from average accuracy to high-confidence predictions at extremely low false-positive rates, they showed that traditional metrics completely mask catastrophic privacy failures affecting small subsets of unique individuals.
This masking effect creates a false sense of security across multiple domains:
- Generative Tabular Data: In A Systematic Evaluation of Generative Models on Tabular Transportation Data, Chengen Wang et al. observed that synthetic datasets can appear perfectly secure on average while still leaking sensitive information from the distribution's extremes. To address this, Xiyang Liu et al. proposed the MACE framework in MACE: A Flexible Framework for Membership Privacy Estimation in Generative Models to calculate sample-level risks, emphasizing that population-wide averages ignore the heightened vulnerability of minority subgroups.
- Machine Learning Unlearning: When attempting to make a model "forget" specific data, average-case evaluations hide lingering risks. Rongzhe Wei et al. demonstrated in Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning that rare data points are more deeply memorized by large language models, making them significantly harder to unlearn and leaving minority populations exposed to higher privacy leakage than standard benchmarks suggest.
The Privacy Onion Effect
A intuitive, naive solution to outlier vulnerability might be to simply identify the most vulnerable data points and prune them from the dataset before training. However, privacy risk in machine learning is a relative property rather than an absolute one.
In The Privacy Onion Effect: Memorization is Relative, Nicholas Carlini uncovered a counterintuitive phenomenon. Peeling away the most vulnerable "outlier" layer of a dataset does not make the remaining model more private. Instead, those extreme outliers were acting as a shield, absorbing the model's memorization capacity and masking nearby points.
Once that outer layer is removed, the model's focus shifts. A new layer of previously "safe" inliers suddenly becomes the new primary target for memorization and membership inference. This relative boundary means that individual privacy vulnerability cannot be engineered away by simple dataset filtering.
Moving Toward Per-Instance Privacy
Recognizing that "one size doesn't fit all"—a concept long understood in genomic privacy fields, as detailed by Sean Simmons et al. in One Size Doesn't Fit All: Measuring Individual Privacy in Aggregate Genomic Data—the machine learning community is shifting toward granular, instance-level privacy paradigms.
Rather than assuming every data point is a worst-case outlier, researchers are analyzing the exact privacy guarantees that individual samples enjoy. In Gradients Look Alike: Sensitivity is Often Overestimated in DP-SGD, Anvith Thudi et al. provided a per-instance analysis of private gradient descent. They proved that because most gradients exhibit high redundancy and "look alike," the majority of data points receive much stronger empirical privacy protections than global bounds suggest. The vulnerability is concentrated strictly in the unique outliers that lack this gradient redundancy.
This insight has paved the way for more adaptive, risk-aware defense mechanisms:
- Attack-Aware Calibration: Instead of injecting uniform noise based on abstract mathematical budgets, Bogdan Kulynych et al. introduced Attack-Aware Noise Calibration for Differential Privacy. This method tunes noise directly to real-world attack risks in low false-positive regimes, recovering massive model utility without sacrificing actual protection for vulnerable points.
- Plausible Deniability Filtering: In Deep Learning with Plausible Deniability, Wenxuan Bao et al. proposed a training framework that uses a rejection-sampling privacy test to discard mini-batches that are too unique to be plausibly denied. By explicitly filtering out updates that carry high individual risk, the algorithm prevents the model from ever learning the extreme outlier signatures that drive privacy leakage.