Federated learning trains one model across many phones and sensors without collecting their data. The textbook version averages weights, so every device must run the same network, and the weakest phone sets the model size for everyone. Federated distillation removes that limit. Each device runs whatever network fits its hardware, labels a shared public dataset with its predictions, and the devices learn from the combined predictions. The cost is that those predictions travel in the clear. A curious server can read them, and a few lying devices can poison them. Existing defenses solve one of these problems while giving up something else. Robust filtering runs in plaintext, cryptographic protection runs only on identical models, and verifiable distillation puts heavy cryptography on the devices themselves.
We propose a different split of the work. Each device splits its prediction table into two random-looking halves, one per server. Two servers that are assumed not to share data with each other then jointly run a robust, outlier-suppressing aggregation over those halves, and only the cleaned consensus is revealed. The device's added cost is one pseudorandom expansion, so its upload is about the same as plaintext distillation. All comparison and exponentiation work runs between the servers. The technical point that makes this cheap is that the nonlinear part of robust distillation can be restricted to one score per client rather than one value per public sample. The cost of the expensive cryptography then grows with the number of clients, not with the size of the public set.
Core Claim
Exponential-weighting defenses for distillation, such as ExpGuard from On the Byzantine-Resilience of Distillation-Based Federated Learning, compute two things: a reference prediction, and a per-client deviation from that reference accumulated over rounds. This structure fits two-server secret sharing well. Additive secret sharing splits a number into two random parts that sum to it, and each part alone reveals nothing. Under this sharing, the reference is a weighted sum. The deviation is a sum of squared differences, which costs one multiplication per coordinate using precomputed multiplication helpers (Beaver triples, correlated randomness the servers prepare offline). Both steps are linear in the size of the public-set prediction table.
The only steps that are expensive under secret sharing are comparisons, clipping and the exponential that turns deviations into weights. These act on $n$ scalars, where $n$ is the number of clients. We evaluate them with function secret sharing, which lets two servers compute comparisons on shared values from compact keys; SIGMA and FuseFSS build these gates for centralized model inference, and we reuse their gate constructions.
The claim we will test has three parts:
- A semi-honest two-server protocol can reproduce ExpGuard-style robust distillation across heterogeneous architectures.
- Its accuracy stays within about one point of plaintext ExpGuard under the published attacks.
- Client upload stays within roughly 10% of plaintext distillation, and server time per round is in minutes rather than the hour scale of proof-based systems.
We also argue that the non-collusion assumption fails gracefully in this setting. If the two servers do collude, they learn individual clients' predictions on public data, which is exactly what plaintext federated distillation already exposes. They do not learn weights or gradients. This is weaker than a guarantee, but it can be measured, and we propose to measure it.
Research Program
The protocol has three roles, and the client's role follows the Aggregator-Compensator pattern of Flimma.
Clients. Each client converts its softmax outputs on the public set into fixed-point numbers. It sends server A the table minus a mask, and sends server B a 128-bit seed from which B regenerates that mask. Flimma uses this pattern only to sum fixed-size linear-regression statistics from identically shaped models. We extend it to function-space outputs that come from arbitrary architectures, and we add a nonlinear robust rule on top.
Servers. Each round runs in three stages.
- Reference. The servers compute a shared reference prediction, starting from the previous round's weights and repeating a reweighting step a small number of times $T$. Each repetition is a shared weight multiplied by a shared table.
- Deviation. For each client $i$, they compute a shared deviation $d_i = \lVert p_i - \bar p \rVert^2$, where $p_i$ is client $i$'s prediction table and $\bar p$ is the current reference.
- Weights. With function secret sharing they update a shared cumulative score $s_i \leftarrow s_i + \min(d_i, \tau)$, clipped at a threshold $\tau$. They compute weights $w_i \propto \exp(-\eta s_i)$ with step size $\eta$, using a piecewise-polynomial exponential, and normalize them.
- Output. Only $\sum_i w_i p_i$ is opened and broadcast. Clients then distill locally against it.
Cost. Online server cost is $O(nPC)$ multiplications plus $O(n)$ function-secret-sharing gates per round, where $P$ is the public-set size and $C$ the number of classes. The alternative of a private coordinate-wise median needs $O(nPC\log n)$ comparisons. For $P{=}10^4$, $C{=}100$ and $n{=}100$, that is six to seven orders of magnitude more comparisons. Avoiding that cost is why we restrict the nonlinear step to per-client scores.
Analysis questions. We will study three questions:
- Fixed-point error. How much precision loss does fixed-point arithmetic and the approximated exponential introduce? SVAFD reports relative error near $10^{-7}$ for its coding scheme, which gives a target.
- Leakage from the output. What does the opened weighted consensus itself reveal? An attacker who controls some clients could difference consecutive rounds to infer another client's contribution. We will measure this leakage and, if needed, open the consensus only to a granularity or with small noise that limits it.
- Leakage under collusion. What can a colluding server pair infer? We will run membership-inference attacks on the revealed public-set predictions and compare them against the same attacks on revealed gradients.
The threat model is stated plainly. The servers are semi-honest and non-colluding. Clients may be malicious, and the robust weighting handles them. We do not claim security against a malicious server. DDFed rejects two-server designs as impractical. We answer that objection in two ways rather than by assumption: by showing that collusion here reduces to an already-accepted exposure, and by reporting the measured cost of the single-server alternative.
Baselines
Naive reference points. The first is plaintext federated distillation with mean aggregation, as in Ensemble Distillation for Robust Model Fusion in Federated Learning. It sets the undefended accuracy floor and the plaintext communication cost. Under a class-prior attack, ExpGuard's paper reports mean aggregation falling to 43.3% on CIFAR-100 and from about 80% to about 46% on CINIC-10. The second naive reference is "do nothing about heterogeneity": every client trains the smallest network under standard secure averaging. HeteroFL reports 58.99% for plain FedAvg against 61.64% for heterogeneous aggregation on non-IID CIFAR-10, which shows the price of a single forced architecture.
Accuracy ceiling. Plaintext ExpGuard is the strongest baseline on robustness. It is the same filtering rule with no privacy, and the paper reports 62.1% on CIFAR-100 under the class-prior attack. Our protocol is judged first by how close it comes to this number. We will rerun ExpGuard ourselves with identical client architectures, public set, attack fraction, round count and hyperparameter search budget, so that the only difference is that the computation is secret-shared.
Strongest existing private competitor. This is SVAFD. It uses Lagrange-coded sharing among client subgroups, filtering based on locality-sensitive hashing, and pairing-based verification. It reports raising accuracy from 36.46% to 61.11% over unprotected distillation under high malice on SVHN and Fashion-MNIST. It also reports client authentication and deblinding under 10 ms in a 1,000-participant network. That figure does not include the peer-to-peer share exchange, which is where we expect the client cost to differ. We will run SVAFD on its own datasets and heterogeneous ResNet-8/16/20 clients. We will report accuracy, total client bytes and client compute separately.
We count as beating SVAFD matching its accuracy within one point while cutting client upload and peer messages substantially. SVAFD does have a malicious-server guarantee and we do not, and the comparison will say so.
Ablations. The first ablation keeps the two-server protocol but opens a plain mean. This is private and non-robust, and it isolates how much the secret-shared weighting buys. The second replaces our per-client scalar filter with a private coordinate-wise median on a reduced public set. It tests whether our cheaper rule gives up robustness. The third ablation varies the number of reweighting repetitions $T$ to measure how much accuracy depends on the iterated reference rather than a single pass.
Why not just…?
Why not just run ExpGuard in plaintext and accept the exposure? Under the semi-honest server model, the server then sees every client's per-sample predictions, and SVAFD argues these reflect local data distributions. The honest answer is that how harmful public-set logits really are is an empirical question. Our membership-inference experiment measures it directly. If the attack is near chance on plaintext logits, the motivation for the protocol shrinks to defense-in-depth, and we will report that.
Why not just add differential-privacy noise to each client's logits? This is the most serious alternative, because logits are bounded and noise added to them is not amplified the way SHE-LoRA reports for LoRA updates. The objection from Private Aggregation for Byzantine-Resilient Heterogeneous FL is that per-client noise blurs honest-versus-malicious distances under non-IID data, so robust filters start discarding honest clients. We will settle this directly. We will sweep the local noise level in noisy plaintext ExpGuard, find the noise at which membership inference on the observed logits drops to the level our protocol leaks, and compare accuracy under attack at that matched leakage. If noisy ExpGuard matches us, local noise is the better engineering choice and we will say so.
Why not just use single-server homomorphic encryption? Encrypted cosine-similarity filtering in DDFed is limited to the last layer of identical models. Proof-based aggregation in zkFL needs about 55 minutes per ResNet-50 update and roughly doubles upload. Homomorphic comparisons and exponentials are the costly part of robust aggregation, and our design moves exactly those to two servers. We will implement an encrypted-mean baseline on logits to report the client-side encryption cost directly.
Why not just use SVAFD? SVAFD gives a stronger server guarantee. Its client-side share exchange within subgroups is a poor fit for intermittently connected phones, and its aggregation is linear. The comparison in Baselines decides whether our cheaper client side is worth the weaker trust model.
Proposed Experiments
Image benchmarks. These experiments use CIFAR-10, CIFAR-100 and CINIC-10, following CINIC-10 is not ImageNet or CIFAR-10. Clients run heterogeneous ResNet-8/16/20 and MobileNet-class networks, data are split non-IID by a Dirichlet draw, and a held-out unlabeled split serves as the public set. The attacks are ExpGuard's loss-maximization and class-prior attacks at 10–40% malicious clients.
Corrupted clients. We test on CIFAR-10-C with 50% corrupted clients. RAHFL reports about 81% against 63% for the vanilla baseline in this setting, which lets us check whether private weighting handles noisy clients as well as RAHFL's plaintext asymmetric transfer.
Metrics. We report global and per-architecture test accuracy, client upload bytes and on-device time, server online and offline time and traffic per round over a wide-area link, and membership-inference success on revealed tables.
Decisive checks. The protocol succeeds only if all four hold:
- Accuracy under the class-prior attack is within one point of our rerun of plaintext ExpGuard. The published reference is 62.1% on CIFAR-100.
- Client upload is within 1.1× of plaintext distillation.
- Server time per round stays in minutes at $n{=}100$ and $P{=}10^4$.
- Membership inference against a colluding server pair is no stronger than against plaintext federated distillation.
If check 1 fails, the cause would be fixed-point error or the exponential approximation, and the ablations will show which. If the differential-privacy comparison favors local noise, the cryptographic design is not justified.
References
- On the Byzantine-Resilience of Distillation-Based Federated Learning
- SIGMA: Secure GPT Inference with Function Secret Sharing
- FuseFSS: Efficient Secure LLM Inference with Function Secret Sharing
- Flimma: a federated and privacy-aware tool for differential gene expression analysis
- SVAFD: A Secure and Verifiable Co-Aggregation Protocol for Federated Distillation
- Dual Defense: Enhancing Privacy and Mitigating Poisoning Attacks in Federated Learning
- Ensemble Distillation for Robust Model Fusion in Federated Learning
- HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
- SHE-LoRA: Selective Homomorphic Encryption for Federated Tuning with Heterogeneous LoRA
- Private Aggregation for Byzantine-Resilient Heterogeneous Federated Learning
- $ \tt {zkFL}$zkFL: Zero-Knowledge Proof-Based Gradient Aggregation for Federated Learning
- CINIC-10 is not ImageNet or CIFAR-10
- Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clients