Fairness Through the Statistics Channel: Private Local-Global Calibration for Federated Adapters

When hospitals or phone users train a shared model without pooling their data (federated learning), two separate problems collide. The first is fairness. A model can be fair to demographic groups inside every hospital and still be unfair across the combined population, which is Simpson's paradox. It can also be the other way round. The second is privacy. The model updates each client sends to the server can be inverted to reconstruct training examples, and the standard defense of adding calibrated noise to every update (differential privacy) hurts accuracy most for small groups. Most attempts to fix both at once put the fairness correction inside the noisy training updates. There, fairness competes with privacy noise for the same scarce signal.

We propose the opposite design. Train a compact adapter under differential privacy with a method already known to survive tight privacy budgets. Then move all fairness correction into a separate channel: a handful of group-by-label prediction counts per calibration round. These counts are summed across clients by secure aggregation, so the server sees only the global total, and they are noised once on that total. Local fairness constraints are enforced on each device using its own exact data, which never leaves. The claim is that fairness information has tiny sensitivity, meaning one person changes one count by one. Spending part of the privacy budget on this channel should therefore close the local-global fairness gap far more cheaply than any in-training fairness method under the same total budget. The approach also costs almost no bandwidth, because only a few dozen numbers are exchanged per round.

Core Claim

Under a fixed end-to-end privacy budget $(\epsilon,\delta)$, a pipeline that combines differentially private bridge-adapter training with secure-aggregated, differentially private post-hoc calibration will match or beat two alternatives. The first is in-training fairness under differential privacy. The second is post-processing with per-client noised statistics. We expect the advantage on both global and local fairness gaps to be largest on data splits where a demographic group is a small minority at every client. The mechanism is a noise-scaling argument. Suppose each of $K$ clients adds Gaussian noise of scale $\sigma$ to its own group counts before sending them, as in local privacy. Then the error in a global count has standard deviation $\sigma\sqrt{K}$. If the noise is added once to the securely aggregated sum, the error is $\sigma$:

$$ \mathrm{sd}\big(\hat n^{\text{global}}_{g,y}\big) = \sigma\sqrt{K}\ \text{(per-client noise)} \quad\text{vs.}\quad \sigma\ \text{(noise on the secure sum)}, $$

where $\hat n^{\text{global}}_{g,y}$ is the released count of examples in group $g$ with label $y$ receiving a given prediction. This matters most when true minority counts are small, exactly the regime where FedFACT and LoGoFair report calibration instability.

Where the Pieces Come From

The fairness side motivates the design. Global Group Fairness in Federated Learning via Function Tracking shows that global demographic parity and equalized odds are not weighted sums of client-level metrics. Minimax Demographic Group Fairness in Federated Learning shows that a group that is a minority everywhere is invisible to client-level heuristics. Mitigating Group Bias in Federated Learning: Beyond Local Fairness bounds global bias by local bias plus a heterogeneity term. Together these say global fairness needs genuinely global statistics, which is why our channel carries aggregate counts rather than local fairness verdicts.

The calibration method is supplied by two post-processing papers. LoGoFair has the server hold dual variables for the global constraint while clients calibrate locally, converging within 10 rounds. The cost of local and global fairness in Federated Learning solves a linear program over summary statistics. We reuse LoGoFair's bilevel structure. We differ in what the server is allowed to see: only a noised secure sum of group-conditioned counts, never per-client statistics. LoGoFair leaves privacy integration open. The cost-of-fairness paper tests only coarse scalar noise and does not study how noise affects the feasibility of its approximation.

The training side is also supplied by prior methods. Fed-SB freezes both low-rank factors and trains only a small square matrix $R$ in $\Delta W = BRA$. It reports 70.33% on SNLI at $\epsilon=1$ against 49.57% for standard federated LoRA. FFA-LoRA freezes one factor and reports 75.05% on MNLI at $\epsilon=1$ against 33.80% for vanilla LoRA. Neither measures group fairness, and we treat that as the gap. Their frozen bases come from random initialization or a small proxy sample, and nothing ensures that minority features live in that subspace.

The leakage side constrains the design. MineGrad shows that a malicious server can turn frozen components and adapter projections into data filters, reconstructing 85.9% of CIFAR-100 patches at batch size 32. Fed-SB's frozen $A$ and $B$ are chosen by the server, so they are exactly such a surface. Our training channel must therefore be protected by differential privacy rather than by its small size. Our statistics channel must be evaluated as an attack surface in its own right, since group-conditioned counts reveal demographic composition, a risk also flagged for the prototypes in Fair-FedMOE.

Research Program

We will build the pipeline in three parts and give them one shared privacy accountant.

In the training phase, clients train Fed-SB bridge matrices with per-example clipping and Gaussian noise.

In the calibration phase, the adapter is frozen and we run LoGoFair-style rounds. In each round the server broadcasts dual variables for the global constraint. Each client applies its current calibrator to its data and enforces its local fairness constraint on-device. Each client then contributes a vector of counts indexed by group, label, and prediction to a secure aggregation, and Gaussian noise is added once to the sum. For $G$ groups in binary classification that is $4G$ numbers per round. The server updates the dual variables from the noised global counts only.

Two refinements address known failure points. The first handles clients whose own minority counts are tiny: they shrink their local constraint estimates toward the released global rates, in an empirical-Bayes fashion. Because this uses only already-released information, it costs no extra privacy. The second is a budget-splitting study, $\epsilon = \epsilon_{\text{train}} + \epsilon_{\text{cal}}$ under standard composition. It asks how small $\epsilon_{\text{cal}}$ can be before calibration degrades.

The analysis side will characterize when the noised dual updates keep the global constraint feasible. We expect this to depend on the smallest true global group count relative to $\sigma$.

Baselines

The naive reference point is differentially private Fed-SB training with no fairness correction, at the full budget $\epsilon$. This tells us whether the gap exists at all under private adapters. If its global and worst-local fairness gaps are already small, the proposal has no problem to solve.

The strongest in-training competitor is FedFACT-In, the cost-sensitive in-processing variant of FedFACT, run with the same differential privacy mechanism on the same Fed-SB adapter and the same total $\epsilon$. On Adult without privacy, FedFACT reports global disparity near zero with about a 2% accuracy drop relative to FedAvg. No privately trained FedFACT result was found on Lacuna. Beating it means lower global and worst-local gaps at equal or better accuracy and equal $\epsilon$.

The strongest post-processing competitor is LoGoFair with each client noising its own statistics before sending, calibrated to the same $\epsilon_{\text{cal}}$. It is our method minus secure aggregation, so it doubles as the ablation that isolates the key ingredient. No reported result for LoGoFair's accuracy or disparity was found on Lacuna, so we will re-run it. On ACSPublicCoverage we will also run the linear-program method of The cost of local and global fairness in Federated Learning with the same noise. That paper reports up to a 98.4% reduction in global disparity versus FedAvg without our privacy setting.

A non-noise competitor is RESFL, adversarial scrubbing of demographic features. It reports a 37% reduction in membership-inference success and a 17% smaller equal-opportunity gap on FACET. It carries no formal guarantee, so we compare it on empirical attack success and fairness, not on $\epsilon$.

A second ablation removes shrinkage. All methods share data splits, backbone, and adapter rank, and each gets the same hyperparameter search budget, with tuning done under privacy accounting or on public validation data only.

Why not just…?

Why not just add noise to LoGoFair's statistics at each client? This is the real threat when there are few clients. In the two-site FairFedMed-Chest setting, the $\sqrt{K}$ penalty is only about 1.4, and trusting secure aggregation buys little. We expect the advantage to grow with $K$ and with minority scarcity. The ablation above measures the gap as a function of $K$ on Dirichlet splits of Adult and ACSPublicCoverage. If per-client noising matches us at realistic $K$, we will report that the simpler scheme suffices.

Why not just train with a fairness regularizer under differential privacy? The fairness gradient signal for minority groups is small and gets clipped and noised along with everything else. RESFL reports that uniform privacy noise disproportionately harms minority subgroups. The privately trained FedFACT-In baseline tests this directly at matched budgets.

Why not just calibrate thresholds centrally after training, with no privacy on the statistics? The released counts leak demographic composition. We will show this with an attack that infers client group proportions from unprotected versus protected releases. Fair-FedMOE acknowledges the same risk for its prototypes.

Why not just rely on the adapter's small size for privacy? MineGrad's results rule this out for LoRA, and the server-chosen frozen bases of Fed-SB make it plausibly worse. We test this with the attack below.

Proposed Experiments

We use Adult, ACSPublicCoverage, ENEM and CelebA with Dirichlet non-IID client splits. We add "minority-everywhere" splits in which one group is under 5% at every client. For the foundation-model setting we use FairFedMed-Oph with a CLIP backbone and Fed-SB adapters, evaluated on race and on age×gender×race intersections.

We report five kinds of metrics:

  • global demographic parity and equal-opportunity gaps;
  • the worst client's local gap;
  • accuracy, or AUC and equity-scaled AUC for FairFedMed, where FairLoRA reports a 2.5% ES-AUC gain;
  • total $\epsilon$;
  • bytes communicated per phase.

We sweep $\epsilon \in \{0.5, 1, 4, 8\}$ and the split between training and calibration.

Three checks are decisive:

  • At $\epsilon=1$ on minority-everywhere splits, our method should reduce the global gap more than both FedFACT-In and per-client-noised LoGoFair while losing no more accuracy.
  • The advantage over per-client noising should grow with $K$ roughly as $\sqrt{K}$, which confirms the mechanism rather than a tuning artifact.
  • An attack check. We adapt PEFTLeak to Fed-SB by letting a malicious server craft the frozen $A$ and $B$, and measure patch reconstruction with and without training noise. We also run a composition-inference attack on the calibration counts. Success rates should fall to near chance at our operating budgets.

If the first check fails, fairness does not belong in a separate low-sensitivity channel, and the note's central claim is wrong.