Who Pays the Privacy Cost of Fair Data Pruning?

Removing a small number of carefully chosen training examples is one of the cheapest ways to make a classifier fairer. Tools called influence functions estimate how much each example pushes the model toward or away from equal treatment of demographic groups. They let practitioners drop the worst few percent of examples, or reweight them, and close fairness gaps with little or no accuracy loss. A separate line of work shows that in federated learning, where devices share only gradients and never raw data, a curious server can often reconstruct private training examples from those gradients. Reconstruction risk is also uneven: some examples are much easier to recover than others. Nobody has asked what fairness pruning does to that uneven risk. Pruning is designed to move the decision boundary toward the protected group and to thin out the examples around it. That same change may leave the remaining protected-group examples isolated in gradient space, which could make them easier to reconstruct.

This note proposes to measure that effect and to remove it if it exists. The core hypothesis is that influence-based fairness pruning shifts reconstruction risk onto the very group it is meant to protect. The remedy is a single reweighting program with three constraints: improve fairness, keep accuracy, and do not raise the protected group's reconstructibility. What makes the remedy possible is a new first-order quantity, the influence of each training example on another group's leakage score. This quantity keeps the reweighting problem a linear program rather than a bilevel search.

Core Claim

The claim has two parts. The first is empirical. On a benchmark that carries both a protected attribute and an image modality, fairness-influence pruning at the 3–40% removal rates reported in the fairness literature measurably raises gradient-inversion success on the protected group. It does so relative to a random pruning at the same rate, and it leaves the other group roughly unchanged.

The second part is methodological. Adding a linearized leakage constraint to an influence-based reweighting program recovers most of the fairness gain at a fraction of the leakage increase. The result should be a better three-way tradeoff than pruning for fairness and then patching privacy with noise or decoys.

If the first part is false, and fairness pruning is neutral or even protective for privacy, that is itself a publishable negative result. It would also make the second part unnecessary. The experiments are ordered so that this is known early.

Background and What This Changes

The fairness side supplies the method. Understanding Instance-Level Impact of Fairness Constraints splits each example's influence into an accuracy part and a fairness part, and shows that pruning by fairness influence reduces violations on CelebA where random pruning does not. Achieving Fairness at No Utility Cost via Data Reweighing with Influence turns these scores into a linear program over example weights and finds a "win-win" set of examples whose removal helps both accuracy and fairness. Fair Infinitesimal Jackknife applies the same correction in closed form using inverse-Hessian products, without retraining. What Data Benefits My Classifier? makes the approach cheap on deep models by computing influence only on the last layer. None of these papers measures privacy, so the intervention's privacy side effects are unknown.

The privacy side supplies the motivating evidence and the leakage score. Understanding Deep Gradient Leakage via Inversion Influence Functions derives a closed-form estimate of how well an example can be reconstructed from its gradient. It calls this estimate I²F, and derives a cheaper lower bound on reconstruction error, I_lb, governed by the model Jacobian's spectrum. The same paper documents "privacy unfairness," meaning that under identical noise some classes stay far more exposed than others. It proposes I²F only as an audit or a training regularizer and never uses it for data selection. We reuse I_lb as the leakage score and move it into the data-curation step.

Two defenses constrain the design. DCS² hides pre-designated sensitive examples behind synthesized decoy gradients, at significant client-side cost and with no fairness accounting. PUFFLE combines a demographic-parity loss with differential privacy in federated learning, but it never tests actual inversion attacks and never touches the data.

DRAUN shows that updates meant to erase examples let a server reconstruct those examples. This is a warning that any data-removal step visible in shared updates is an attack surface. For that reason we apply reweighting locally, before training updates are shared, and not as a transmitted correction.

Research Program

The setting is cross-device federated learning simulated on a single machine. Clients hold small local datasets and share per-step gradients at batch sizes of 1 to 16, where inversion is known to work. An honest-but-curious server runs an optimization-based gradient inversion attack.

The first stage is an audit. We compute fairness influence using the decomposition of Understanding Instance-Level Impact of Fairness Constraints, with last-layer approximations where needed, and prune at several rates. We then retrain the federated model and measure reconstruction quality for each group, as structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) between true and recovered images. We also measure I_lb for each group, together with accuracy, demographic parity gap, and equal opportunity gap.

We track one mechanistic diagnostic throughout. For protected-group examples, we record the gradient's nearest-neighbour distance and the leading Jacobian singular value, to test whether rising leakage coincides with rising gradient isolation as the gap analysis suggests.

The second stage is the remedy. Let $w_i \in [0,1]$ be the weight on training example $i$. Let $f_i$ be its first-order influence on the fairness gap, $u_i$ its influence on validation loss, and $\ell_i$ its influence on the protected group's mean leakage score. The leakage score is defined as the group's average of $-I_{lb}$, so that larger means easier to reconstruct. We solve

$$\min_{w} \sum_i (1-w_i) \quad \text{s.t.} \quad \sum_i (1-w_i) f_i \le -\Delta_F,\;\; \sum_i (1-w_i) u_i \le 0,\;\; \sum_i (1-w_i)\, \ell_i \le \tau,$$

where $\Delta_F$ is the targeted fairness-gap reduction and $\tau$ is the allowed increase in protected-group leakage, with $\tau = 0$ as the default.

The new ingredient is $\ell_i$. It is the inner product between the parameter shift caused by downweighting example $i$ (the standard influence term $H^{-1}\nabla \ell(z_i)$, where $H$ is the training-loss Hessian) and the gradient of the protected group's leakage score with respect to parameters. We obtain the leakage-score gradient by differentiating the top Jacobian singular value through its power-iteration singular vectors. All three constraints are therefore linear in the weights, as in Achieving Fairness at No Utility Cost via Data Reweighing with Influence.

We expect the linear approximation to degrade when many examples are removed together, the known group-effect problem. Two measures address this. First, the program is re-solved over several rounds with recomputed scores. Second, we report how well predicted shifts in the leakage score correlate with shifts measured after retraining.

Baselines

The naive reference is the unpruned federated model trained on all data. It fixes the starting fairness gap and the starting per-group reconstruction quality. Random pruning at matched removal rates is the second reference, because it separates the effect of simply having less data from the effect of choosing which data. The fairness literature already reports that random pruning leaves bias intact on CelebA, so it should lose on fairness. Its leakage numbers are the control for the audit.

The strongest fairness competitor is fairness-only influence reweighting, meaning the same linear program without the leakage constraint, which reproduces Achieving Fairness at No Utility Cost via Data Reweighing with Influence. We pair it with closed-form correction via Fair Infinitesimal Jackknife. Fair-IJ reports reducing the BERT demographic parity gap on CivilComments from 0.246 to 0.011. On CelebA, the fairness-pruning paper reports qualitative trends; no reported scalar result was found on Lacuna for fairness pruning on CelebA. All methods use the same model, the same federated schedule, the same validation split for influence, and the same number of influence recomputations. Beating this baseline means matching its fairness gap within its seed variance while lowering protected-group reconstruction SSIM by a clear margin at equal accuracy.

The strongest privacy competitor is pruning for fairness followed by DCS² applied to protected-group examples. DCS² reports reconstruction SSIM of 0.15 on CIFAR-10, and PSNR of 7.84 against 59.20 undefended on MNIST. It is run with its published settings at matched client compute. Beating it means equal or better protected-group SSIM at lower client compute and with no fairness loss.

A differential-privacy baseline in the style of PUFFLE is also included. PUFFLE reports up to 75% unfairness reduction with at most 17% accuracy loss. This baseline is evaluated against the actual inversion attack, not only by its privacy budget.

The key ablation drops the cross-group term. It replaces $\ell_i$ with each example's own I_lb, pruning the examples that are most reconstructible themselves. This tests whether influence on other examples' leakage matters, or whether a per-example leakage filter suffices.

Why not just…?

Why not just add differential-privacy noise after pruning? Noise does attack reconstruction directly. However, Understanding Deep Gradient Leakage via Inversion Influence Functions shows that uniform noise leaves some classes exposed while over-protecting others. DRAUN and PUFFLE both report large accuracy costs, which compound with the group disparity that noise induces. A real threat is noise calibrated per example using I_lb, which the I²F paper effectively suggests. We include it as a baseline: if calibrated noise matches our protected-group SSIM at equal accuracy and fairness, the reweighting adds nothing and we will say so.

Why not just prune by each example's own leakage score? This is the simplest combination: prune for fairness, then also drop the most reconstructible examples. It is the main ablation. The hypothesis is that leakage rises because pruning reshapes the model around the remaining minority examples. Removing more minority examples would then worsen fairness and could isolate the rest further. If the self-score filter performs as well as the cross-group term, the core mechanism is wrong.

Why not just use larger local batches? Inversion collapses at batch sizes of 64 and above, so this is a cheap defense. Cross-device clients often hold too little data for large batches, and small batches are where the leakage literature finds real risk. We report every result as a function of batch size. If the fairness-induced leakage increase disappears by batch size 16, the problem is narrow, and we will scope the claim accordingly.

Proposed Experiments and Decisive Checks

CelebA is the primary benchmark because it appears in both literatures. We use it with "Smiling" as the target, gender as the protected attribute, and ResNet-18 clients under non-IID partitions, attacking at batch sizes 1, 4 and 16. Adult and COMPAS serve as tabular checks, using an MLP and optimization-based feature reconstruction, so that the fairness numbers can be compared to the reweighting literature. CivilComments with BERT tests whether the audit finding carries over to text, using recovery overlap scores.

The first decisive check is the audit. It asks whether the protected group's attack SSIM rises under fairness pruning, compared with random pruning at matched rates, across five seeds. If not, the second stage is dropped and the paper reports the null result.

The second check is predictive validity. The linearized leakage influence must correlate with leakage shifts measured after retraining. Fairness influence reaches Pearson correlations above 0.99 on Adult and COMPAS in the reweighting paper, so the bar is meaningful agreement well above chance, reported honestly if it is weaker for leakage.

The third check is the three-way frontier. Accuracy, fairness gap, and protected-group SSIM are plotted together against every baseline above. This shows whether the joint program contains points that dominate pruning followed by DCS² or by calibrated noise.