Fairness-Aware Noisy Voting for Private, Low-Bandwidth Federated Distillation

Federated distillation lets phones, hospitals or other data holders train a shared model without sending their model weights. Instead, each participant labels a shared pool of unlabeled "public" examples with its own model and sends back only those predictions. This makes messages tiny and lets every participant use a different architecture. It was also assumed to be private. That assumption has broken down. An attacker who compares a client's predictions with the server's can reconstruct the client's private training images. Separately, the standard defense of adding random noise (differential privacy, a guarantee that no single contributor's data changes the output much) is known to hurt small demographic groups the most. Their signal is weak, so noise erases it first.

We propose a distillation protocol in which each client sends one bit per public example, namely its predicted class. The server counts these votes with calibrated noise, which gives a formal privacy guarantee at the level of whole clients. The central idea is to spend that privacy budget unevenly across public examples. More budget, and therefore less noise, goes to the examples where clients disagree and that resemble underrepresented groups. The server learns which examples those are only through a second noisy one-bit vote whose group labels never leave the client. The bet is that the tension between privacy and fairness in federated distillation lives on a small, identifiable slice of the public set. If so, targeting that slice lets a single low-bandwidth channel resist reconstruction, carry a formal guarantee, and still teach minority groups well.

Core Claim

In federated distillation, three concerns meet in one object: the per-example prediction a client sends about the public set. The communication cost is its bit-width. The reconstruction attack in Breaching FedMD: Image Recovery via Paired-Logits Inversion Attack works by exploiting the "confidence gap" between client and server soft predictions. Fairness depends on whether the aggregated teacher is right on public examples resembling minority subgroups. We make two claims.

First, cutting uploads to a single vote removes the confidence values that this reconstruction attack relies on. It also makes the per-client influence on the aggregate exactly bounded, so private aggregation in the style of noisy majority voting becomes almost free in bandwidth.

Second, uniform noise on those votes produces a fairness failure specific to distillation. Public examples near minority subgroups are exactly the ones with split votes. Split votes are the ones that noise flips, so the student model is taught wrong labels there. Allocating the privacy budget toward low-consensus, minority-proximal public examples should recover most of the fairness gap at a fixed privacy cost.

The second claim is the new part, and it is falsifiable. If uniform noise plus a local fairness regularizer already closes the gap, the allocation step is unnecessary. We will report that result.

Relation to Prior Work

Several papers motivate the gap. On communication, CFD shows that federated distillation survives quantization down to 1-bit class indices with round-to-round delta coding. It reports more than four orders of magnitude less traffic than federated averaging. ReT-FHD reaches 0.0003 MB per round with temperature calibration and a Z-score guard against poisoned logits. Neither paper measures group fairness or reconstruction risk. We reuse CFD's 1-bit channel as our transport and ask what it does to both.

On privacy, the paired-logits inversion attack rebuts the claim in Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions that logit sharing blocks reconstruction. Yet no defense has been evaluated against it. Gradient-Leakage Resilient Federated Learning (Fed-CDP) defends parameter-sharing training with per-example clipping and noise, at a per-example compute cost. It also inherits the regressive effect of noise on small groups that RESFL documents.

On fairness, several works constrain our design. FairFed reweights clients using fairness statistics gathered by secure aggregation. Global Group Fairness in Federated Learning via Function Tracking shows that local fairness does not imply global fairness. That paper also coordinates clients through roughly 50 broadcast scalar anchor predictions per group, and it shows that noise on predictions acts like a kernel smoothing. RESFL weights clients by predictive uncertainty and needs no server-side sensitive labels. All three still exchange full model parameters. They are therefore exposed to gradient inversion and tied to one shared architecture.

We borrow two ideas from these papers. From function tracking we take the observation that fairness coordination can run through a handful of prediction-level statistics. From RESFL we take the use of disagreement as a stand-in for poor representation.

The local fairness regularizer comes from Logits-MMD. That method aligns the distributions of model outputs across groups using maximum mean discrepancy, a kernel distance between distributions, but it has only been studied centrally. FairDD shows that compressing data into a few synthetic points collapses minority subgroups. It concerns dataset distillation, not federated distillation, but it warns that aggressive compression, including our 1-bit votes, may do the same. That is the failure mode our allocation step targets.

Research Program

Protocol. Each round has four steps.

  1. Every client trains locally with cross-entropy plus a Logits-MMD penalty computed on its own group labels. The group labels never leave the device.
  2. The client casts two sets of one-bit votes on a sampled batch of public examples. The first is its predicted class. The second is a group-proximity vote: for each protected group $g$, a flag saying whether this public example's representation falls within the client's nearest-neighbour radius of its own group-$g$ training samples.
  3. The group-proximity votes are summed under secure aggregation with Gaussian noise. This gives the server a noisy per-example minority-proximity score. The class votes are then aggregated by noisy argmax.
  4. Only public examples whose consensus passes a noisy threshold are released as teacher labels. The noise scale for example $i$ is set by the allocation rule below.

Allocation rule. The noise standard deviation $\sigma_i$ for example $i$ is chosen to minimize the expected number of wrong released labels, weighted by $1+\lambda m_i$. Here $m_i$ is the example's noisy minority-proximity score and $\lambda$ trades fairness against overall accuracy. The optimization is subject to a total per-round privacy cost. We account for that cost with the standard data-dependent analysis of noisy vote aggregation, composed across rounds.

The guarantee is client-level: changing one client's entire dataset changes each vote count by at most one. That is stronger than the per-example guarantee of Fed-CDP. The price is that client-level guarantees need many clients, so we will report results as a function of client count.

Analysis. We will measure, per public example, the confidence gap exploited by the reconstruction attack, the vote margin and the minority-proximity score. The test is whether reconstruction risk and fairness loss are concentrated on the same slice of examples, which is the hypothesis behind the whole design. We will also check whether the group-proximity vote itself leaks group membership. To do so, we attack it with a membership-style inference attack and require its success to stay near chance at the chosen noise level.

Baselines

The naive reference point is plain federated distillation with full-precision soft labels averaged by the server, as in FedED. It has no noise and no fairness term. It fixes the utility ceiling and the worst-case exposure to reconstruction.

The strongest existing low-bandwidth competitor is CFD with 1-bit quantization and delta coding. It is the published system closest to our transport. We run it on identical client splits, the same public pool, the same architectures and the same round budget, and tune its hyperparameters with the same number of trials we give our method. CFD reports a communication reduction of more than 10,000x relative to federated averaging. We do not need to beat that. We need to match its bits per round within a small constant factor while improving the fairness gap and reconstruction resistance.

The strongest fairness-and-privacy competitor is RESFL. On FACET it reports mAP 0.6654, a 37% reduction in membership-inference success rate and a 17% reduction in the equality-of-opportunity gap, both relative to federated averaging. We rerun RESFL on the same FACET splits and compute budget. Beating it means matching its equality-of-opportunity gap within its variance across seeds at comparable mAP, while sending orders of magnitude fewer bits and supporting mixed architectures, which RESFL cannot. We also run FairFed and parameter-sharing training with client-level noise plus FairFed weighting as parameter-based references. No reported result was found on Lacuna for any federated distillation method on FACET or on Adult with a fairness metric, so those numbers will be produced here.

Two ablations isolate the key ingredient. The first uses uniform noise across public examples with the same total privacy cost and keeps the local Logits-MMD penalty. The second uses the allocation rule but drops the local regularizer. Both run under the same accounting.

Why not just…?

Why not just use CFD's 1-bit votes, since quantization may already defeat the inversion attack? It may well blunt the attack. Paired-logits inversion trains on confidence gaps, and argmax votes have none. That is still an empirical defense with no guarantee, and repeated rounds of votes could leak information through changes over time. We run the attack, adapted to vote histories, against CFD. If CFD alone keeps reconstruction quality near that of a random baseline, we will say so. The contribution then narrows to the fairness allocation and the formal guarantee.

Why not just add Logits-MMD locally and use uniform noise? This is the real threat, and it is our first ablation. Our prediction is that a local regularizer cannot repair the teacher labels that noise flips on split-vote examples. FairDD's collapse result under heavy compression makes this plausible. If the uniform-noise ablation matches the full method on equality-of-opportunity gap at equal privacy cost, the allocation step fails, and we report that.

Why not just run FairFed or RESFL on top of noisy parameter-sharing training? Those methods send full parameters. They therefore stay exposed to gradient inversion, cost as much bandwidth as federated averaging, and require identical architectures. RESFL also reports membership inference rather than reconstruction. Our comparison measures all three axes on the same data so the trade can be read directly.

Proposed Experiments

We use FACET, with Monk Skin Tone as the protected attribute. We split it into client shards that are skewed both in label and in demographics, plus a held-out public pool stripped of attributes and labels. We use Adult across silos defined by region, with gender and race as protected attributes, as a tabular check. We use the CARLA fog and low-light scenarios as a stress test of pedestrian detection across skin tones; there we report only relative comparisons, since the file lists no numbers for CARLA.

Metrics are:

  • utility: mAP on FACET and accuracy on Adult;
  • fairness: the equality-of-opportunity gap and the worst-group accuracy;
  • reconstruction: SSIM and the attack's identification rate, the fraction of reconstructions matched to the correct private individual;
  • membership inference: attack success rate;
  • privacy cost: the reported $(\epsilon,\delta)$;
  • bandwidth: bits uploaded per client per round.

Clients use mixed architectures.

The decisive checks are four. First, the reconstruction attack's success should fall to near a random baseline under our protocol. Second, the correlation analysis should show fairness errors concentrated on low-margin, minority-proximal public examples. Third, at equal $\epsilon$, allocation should beat uniform noise on the equality-of-opportunity gap. Fourth, the full method should reach RESFL's fairness level at a small fraction of its bandwidth. Failing the second check would falsify the premise of the proposal.