Federated learning lets many devices train a shared model without sending their raw data anywhere. The standard recipe averages model weights, and that fails when phones, microcontrollers and workstations run different networks. The usual workaround is federated distillation: clients exchange predictions instead of weights. That workaround needs a shared public dataset to make predictions on, and it offers no formal privacy guarantee. Both gaps matter most where federated learning matters most, in hospitals and banks, which have no public data to share and face real reconstruction attacks.
This proposal combines four pieces so that only real-data training steps cost privacy. Each client trains its own architecture with differentially private stochastic gradient descent (DP-SGD), a training rule that clips each example's gradient and adds calibrated noise so the trained model provably reveals little about any single record. The server then trains a generator against these privatized models and sends synthetic samples back. By the post-processing property of differential privacy (anything computed from a private output is equally private), the generator, its samples, and every distillation step clients take on those samples cost no extra privacy budget. Clients can therefore take few, expensive private steps on real data and many free steps learning from the federation's synthetic consensus. We expect this to help most for the small models on weak devices, which suffer worst from DP-SGD's noise and clipping.
Core Claim
In data-free federated distillation, the privacy cost should be charged only at the client's real-data optimizer and nowhere else. Moving local computation from real-data steps to synthetic-data distillation steps should give mixed-architecture federations a much better accuracy-privacy trade-off than either private weight averaging or private local training alone. It should do so without any public dataset, and it should reduce the extra accuracy loss DP inflicts on minority classes and on low-capacity clients.
The accounting is the whole point. For client $k$, let $q_k$ be its sampling rate, $\sigma_k$ its noise multiplier, and $T_k^{\text{priv}}$ the number of DP-SGD steps it takes on its own data over all rounds. Its record-level guarantee is
$$\varepsilon_k = \mathcal{A}\big(q_k, \sigma_k, T_k^{\text{priv}}, \delta\big),$$
where $\mathcal{A}$ is a standard composition accountant. The number of synthetic distillation steps $T_k^{\text{syn}}$ does not appear in this expression. The reason is that those steps consume only quantities computed from other clients' already-privatized models and from the client's own privatized model.
Position Relative to Prior Work
Three families of work each cover part of this space. Public-data private distillation is the first. FedMD-NFDP gets privacy from random subsampling of local data while clients share predictions on a public set. FedAUXfdp pretrains a frozen feature extractor on public data and privatizes only a logistic-regression head. Both give formal DP across heterogeneous models, but both need public data. FedAUXfdp's frozen extractor also cannot learn new features from private data, whereas we train every layer.
Data-free federated distillation is the second family. FedGen trains a server-side conditional feature generator. DFRD keeps an exponential-moving-average generator to fuse heterogeneous clients without proxy data. Both remove the public-data dependency, and both papers acknowledge that the generator may encode client distributions while offering no guarantee. We reuse DFRD's server mechanism almost unchanged. The difference is that it consumes only DP-trained client models, which turns their acknowledged leak into a bounded one.
DPDFD is the closest method. It distills a sensitive teacher into a student through a generator, with gradient normalization and low-dimensional output noise. It is a centralized, single-teacher conversion and does not address multi-round federations of differently shaped clients.
Two lines of work motivate the design rather than supply it. The first is generative gradient inversion. GIAS and GIML reconstruct training images at batch size 32 and survive pruning of 99% of gradient entries. They can even meta-learn the image prior from streamed updates, which tells us heuristic defenses on the bridge are not enough. The second is Bagdasarian et al., who show that DP-SGD taxes minority groups. For example, a minority digit fell from 97% to 77% accuracy on MNIST. That result constrains the design: whatever we do must be measured per class and per device tier, not only on average.
HeteroFL and FjORD handle heterogeneity through nested width slices of one model family. They are a competitor for the width-only setting, but they cannot mix convolutional networks with attention models.
Research Program
The protocol runs in rounds. Each client trains its own architecture with DP-SGD for a small, fixed number of private steps. Candidate architectures are ResNet-20/32, ShuffleNetV2 and a small vision transformer. The client then uploads its privatized model. White-box access is acceptable because the uploaded model is itself DP.
The server trains a class-conditional generator, DFRD-style, to produce samples on which the client ensemble agrees confidently and disagrees informatively. It also distills the ensemble into a server student. It then broadcasts the generator, not the student, and each client takes many local distillation steps on generated samples against ensemble soft labels. These distillation steps are free under the accounting above.
We will study three design questions. The first is the split between private and synthetic steps at a fixed $\varepsilon$, which is the proposal's central knob. The second is tier-dependent clipping thresholds and noise multipliers, where small models get a clip value matched to their gradient-norm scale. This addresses the mismatch the research file raises between one fixed clip bound and subnetworks of very different size, and it is legitimate because each client's guarantee is accounted separately. The third is whether conditioning the generator on class labels concentrates synthetic samples on the minority classes that DP-SGD starves. That would directly counter the regressive tax.
Alongside the method we will run a leakage audit, which is the gap the research file identifies for data-free bridges. GIAS-style inversion will be aimed at the generator's updates and samples in non-private FedGen and DFRD, and at the same components in our protocol. We will also run membership inference against generator samples. The guarantee makes the outcome for our protocol predictable in principle, so the audit serves two purposes. It checks the implementation and accounting, and it shows concretely what the non-private data-free methods expose.
Baselines
The naive reference point is isolated private training: each client runs DP-SGD on its own architecture and data at the target $\varepsilon$ with no communication. Every collaborative method must beat it per tier, and it shows how much the federation adds. A second naive reference is privatized FedAvg on a single shared architecture. Hoech et al. report it reaching only 46.0% accuracy on CIFAR-10 with ShuffleNet at Dirichlet $\alpha=0.01$.
The strongest existing method with a formal guarantee is FedAUXfdp. Hoech et al. report 75.2% accuracy in that same CIFAR-10 setting (ShuffleNet, $\alpha=0.01$), with typically under 1% loss at $\varepsilon=0.6$ relative to non-private training. It is the strongest competitor because it is the best-documented private heterogeneous distillation method in the file. Its advantage is public data. We will run it with its published public-data setup at matched $\varepsilon$, $\delta$, client count and partition. We will also run it with out-of-domain public data, to measure how much its number depends on that resource. FedMD-NFDP, at 74.40% on CIFAR-10 with $k=16$ and 96.63% on FEMNIST as reported by Sun & Lyu, is the second public-data competitor and will be run the same way.
Beating them has a specific meaning here. We count it a success if, using no public data, our method matches or comes within a small margin of FedAUXfdp at equal $\varepsilon$ on CIFAR-10. It must also clearly exceed privatized FedAvg and isolated training, and it must keep the per-class gap no larger than theirs. For the width-nested setting we add HeteroFL with DP-SGD at matched $\varepsilon$ and tuning budget.
The ablation that isolates the key ingredient is our own protocol with the private-to-synthetic split fixed at one-to-one. A second ablation sets the synthetic steps to zero: the server still distills, but clients receive nothing back. Gains over these ablations, at equal $\varepsilon$ and equal wall-clock compute, measure what free synthetic steps contribute. Every method gets the same hyperparameter search budget. Each search is either counted against the privacy budget or done on a held-out proxy split, following the extrapolation approach of Koskela et al..
Why not just…?
Why not just train a DP generator once and use its samples as the public proxy for FedDF or FedMD? This is the most serious threat. It is simpler and also exploits post-processing. The difference is that a one-shot private generator is trained before any client model is good, so its samples cannot target where the current ensemble is uncertain. Our generator is re-trained each round against the improving private models. Whether that matters is an empirical question, and we will settle it directly. The two-stage baseline will spend the same total $\varepsilon$, split between generator training and local training, and use the same architectures. If it matches us, the right conclusion is that iterative bridging is unnecessary, and we will report that.
Why not just run DP-SGD in plain FedAvg? It requires identical architectures and so excludes the setting we target. Even in the homogeneous case, Hoech et al. report that it collapses to 46.0% under strong label skew.
Why not just rely on an out-of-domain public dataset? Lin et al. found that out-of-domain transfer can work for natural images. Wu et al. show that distant proxies yield high-entropy, uninformative pseudo-labels, and in pathology no such proxy may be lawful to use. Our out-of-domain FedAUXfdp runs on PathMNIST measure exactly this. If the proxy works well there, the motivation for being data-free weakens for that domain.
Why not just defend the bridge with sparsification or larger batches? GIAS reconstructs inputs at batch size 32 and under 99% pruning, and GIML learns its prior from the stream itself. The leakage audit will reproduce this against unprotected data-free generators.
Proposed Experiments
The first experiment is CIFAR-10 under Dirichlet partitions ($\alpha \in \{0.01, 0.1, 1\}$) with mixed ResNet, ShuffleNetV2 and small-transformer clients. It sweeps $\varepsilon$ from 0.5 to 8 and reports global and per-tier accuracy. CIFAR-100 serves as a harder variant. FEMNIST from LEAF supplies natural writer-level skew.
PathMNIST is the sensitive-domain test. The only reported private number in the file there is centralized: DPDFD reaches 80.12% at $\varepsilon=1$. We treat it as a reference ceiling rather than a federated baseline. No reported result was found on Lacuna for any federated private method on PathMNIST.
For disparate impact we follow Bagdasarian et al. and reproduce their artificial-minority setup on MNIST and CIFAR-10. We assign minority classes preferentially to low-capacity clients, which is the compounded case raised in the research file, and report minority-class accuracy against the isolated and privatized-FedAvg references. CelebA is used to report subgroup gaps.
The leakage audit reports PSNR from GIAS-style attacks. As a reference, Jeon et al. report 15.58 dB for GIAS against 13.78 dB for pixel-space attacks under FedAvg. The audit also reports membership-inference advantage on generator samples, for FedGen, DFRD and our protocol.
Three outcomes would be decisive. The first is the accuracy curve as $T^{\text{syn}}$ grows at fixed $\varepsilon$: if it is flat, the core claim fails. The second is the head-to-head against the one-shot DP-generator proxy. The third is whether small-model clients gain more than large ones from free synthetic steps, which would confirm the mechanism behind the fairness claim.
References
- Federated Model Distillation with Noise-Free Differential Privacy
- FedAUXfdp: Differentially Private One-Shot Federated Distillation
- Data-Free Knowledge Distillation for Heterogeneous Federated Learning
- DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning
- Model Conversion via Differentially Private Data-Free Distillation
- Gradient Inversion with Generative Image Prior
- Differential Privacy Has Disparate Impact on Model Accuracy
- HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
- FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
- Practical Differentially Private Hyperparameter Tuning with Subsampling
- Ensemble Distillation for Robust Model Fusion in Federated Learning
- Communication-efficient federated learning via knowledge distillation
- LEAF: A Benchmark for Federated Settings