In federated learning, many phones or hospitals train one shared model without sending their raw data. Each client sends back a model update instead. Two separate lines of work have found that the smallest piece of that update is the most revealing one. That piece is the change to the final classification layer, the C numbers of the output bias plus the C×d output weights, where C is the number of classes. One line uses it constructively. HiCS-FL reads how balanced each client's labels are from the entropy of its bias update and favors balanced clients early in training, reaching target accuracy 7.3× faster than random sampling on CIFAR-10. The other line uses the same signal as an attack. Decaf reads off which classes a client lacks with 100% accuracy and recovers its class proportions to within 5% in the largest per-class error. The server that HiCS-FL makes faster is exactly the server that Decaf shows can profile every client.
This note proposes a protocol that separates the two uses. The server receives only an order-independent summary of each client's label balance, which is all it needs to pick clients. The class-identifying final layer is summed under cryptographic secure aggregation, which is cheap here because that layer is tiny. The large feature-extractor update stays on an aggressively sparsified uplink. The work has two parts. We build this split uplink, and we measure whether compression of the feature-extractor update leaks label information through the backward chain shown by Building Gradient Bridges.
Core Claim
Client selection driven by label heterogeneity does not require the server to know which classes a client holds. It requires only a quantity that ignores class identity, such as the entropy of the client's label distribution. We claim three things.
First, an entropy-only selection rule fed by a locally computed, locally noised scalar keeps most of HiCS-FL's speedup.
Second, the final layer can be aggregated under secure aggregation at negligible bandwidth cost. Secure aggregation is a cryptographic protocol in which clients add masks that cancel in the sum, so the server sees only the total. Its cost is negligible here because the final layer is a small dense block. It is also the only practical place to use it. Pairwise masks make vectors dense, so secure aggregation cannot be combined with Top-k sparsification of the full model, which sends only the k largest entries.
Third, once the final layer is hidden, the remaining leakage comes from the sparsified feature-extractor update. Whether high sparsity (above 99%, with error feedback) cuts the gradient-bridge chain is an open empirical question that we will settle directly. GDBR showed the chain survives 50–90% pruning, but no one has tested the regime communication-efficient training actually uses.
The threat model is a passive, honest-but-curious server that holds the global model and a small auxiliary labeled set, as in Decaf. We do not defend against an active server that distributes manipulated models. Breaking Secure Aggregation shows that such a server can tamper with batch-normalization parameters, disaggregate a securely summed update, and recover labels perfectly. We state this limit plainly rather than hide it.
Research Program
Measurement study. We first pin down how standard compression operators change the final-layer signal that both HiCS-FL and Decaf rely on. The cross-entropy bias gradient is $p_c - y_c$, where $p_c$ is the predicted probability of class $c$ and $y_c$ is the one-hot label. Its sign is negative for present classes and positive for absent ones, as documented in Label Leakage in Vertical Federated Learning: A Survey.
We will test four operators on multi-step local training (FedAvg) rather than single-step FedSGD:
- Top-k with error feedback, which keeps unsent entries in a local residual and sends them later.
- Sign compression.
- Stochastic 8-bit quantization.
- The layer-wise Mixed Quantization scheme, which keeps scale factors private.
Our hypothesis is that sign-preserving schemes keep null-class leakage fully intact. We also expect Top-k with error feedback to leak through timing: small positive drifts on absent classes are delayed but eventually sent. If either effect holds, quantization defenses tuned against pixel reconstruction say nothing about distribution privacy.
Protocol construction. Each round, a selected client computes the entropy $H$ of its local label histogram and reports
$$\tilde H = H + \text{Laplace}(\Delta_H/\varepsilon),$$
where $\Delta_H$ is the sensitivity of entropy to changing one sample for a client with at least $n$ samples (on the order of $\log n / n$) and $\varepsilon$ is the local privacy budget. This is a single scalar per client, so the extra communication is negligible.
The server runs HiCS-FL's annealed schedule using only the entropy-difference term. It drops the bias-direction similarity term, which requires class identity. Final-layer updates go through secure aggregation restricted to those C(d+1) coordinates. Feature-extractor updates are Top-k sparsified with error feedback at rates from 90% to 99.9%. Optionally, the client applies a variant of Soteria-style pruning to the penultimate representation before sending.
Analysis. Selection is now blind to which classes clients hold, so we will characterize when this costs convergence. Two clients with identical entropy but disjoint classes are interchangeable to our rule but not to HiCS-FL's clustering. We expect the loss to be concentrated in partitions where clients are pathologically disjoint, for example two classes per client. We will report it separately there.
Baselines
Random sampling with FedAvg. This is the do-nothing reference for both speed and privacy. The server sees uncompressed updates and picks clients uniformly. It anchors rounds-to-accuracy and uplink-bytes-to-accuracy. Decaf and GDBR run against it to give the unprotected leakage level.
HiCS-FL. HiCS-FL is the strongest constructive competitor because it already achieves heterogeneity-aware selection at zero extra parameter communication. Its reported results are 7.3× faster than random on CIFAR-10 and 2.5× on FMNIST. We run it with its published schedule on identical partitions, models, and local-epoch settings, with the same hyperparameter search budget as our method. We also run it on Top-k-compressed uplinks, which its paper does not evaluate. Beating it means matching its rounds-to-accuracy within 15%, cutting Decaf's attack to near-chance, and transmitting fewer uplink bytes.
FedCG. FedCG is the strongest compression-plus-selection system. It picks diverse clients by maximizing a submodular coverage objective over historical gradients and assigns per-client Top-k ratios by linear programming. It reports 4.9× hardware speedup on CIFAR-10 and 50.3% traffic reduction. It shares our compression regime, so we match its total uplink budget per round. Because its selection inspects per-client gradients, we also attack it with Decaf and GDBR.
Mixed quantization as a privacy-communication competitor. Mixed Quantization reports DLG reconstruction quality dropping from 55.3 to 8.1 PSNR (peak signal-to-noise ratio) at 4× compression. Here it is a defense whose effect on distribution inference is untested.
Ablations isolating the key ingredient. The first keeps HiCS-FL's full signal but hides the final layer under secure aggregation. This measures the speed cost of losing per-client bias directions. The second keeps our entropy-only selection but sends the final layer in the clear. This shows whether secure aggregation of the head, rather than selection, is what defeats Decaf.
Why not just…?
Why not just add differential privacy noise to the whole update? Differential privacy here means adding calibrated random noise so that no single sample changes the output much. The file's evidence is consistent: uniform noise strong enough to stop label recovery wrecks utility. Towards Eliminating Hard Label Constraints recovers labels until noise variance reaches 10⁻¹. Decaf reports differential privacy only degrades its attack at significant utility cost under non-IID data. Mixed Quantization reports roughly 78–85% MNIST accuracy for its differential-privacy baseline versus 96.67% for its scheme. We nonetheless include a head-only differential-privacy variant at matched accuracy as a direct comparison.
Why not just have clients send a noised label histogram instead of an entropy? This is the real threat to our design, since a histogram would let the server keep HiCS-FL's clustering. But a histogram has C coordinates that each name a class. Under the same privacy budget it needs far more noise per coordinate than one scalar, and it reintroduces exactly what Decaf extracts. We will run noised-histogram selection at equal ε. If it beats entropy-only selection on convergence without measurably more attack success, we adopt it and the claim narrows to "the final layer must be hidden."
Why not just withhold the final layer, or rely on heavy sparsification? GDBR shows that withholding the final layer still yields over 80% label recovery, over 95% on ImageNet, and survives 50–90% pruning. Evaluating Gradient Inversion Attacks and Defenses needs over 99% pruning to stop pixel reconstruction, at roughly 10% accuracy loss. Our protocol does not assume sparsification is a defense. It measures GDBR at 99%-plus sparsity with error feedback, and that measurement decides whether the Soteria-style add-on is needed.
Why not just use secure aggregation on everything? Masked vectors are dense, so full secure aggregation forfeits Top-k savings. It also blinds the server entirely, forcing random sampling, which HiCS-FL shows can be up to 7.3× slower.
Proposed Experiments
Datasets and partitions. We use FMNIST, CIFAR-10, CIFAR-100, and THUCNews, which match the HiCS-FL and Decaf evaluations. Partitions are label-skewed, either Dirichlet or two-class-per-client, with 100 clients and 10 sampled per round. Models are a small CNN and ResNet-18, plus VGG16 to probe Decaf's reported weakness with high-dimensional heads.
Metrics. Utility is measured by rounds and uplink bytes to target accuracy. Privacy is measured by three numbers: Decaf null-class accuracy and largest per-class error, GDBR label-recovery accuracy, and auxiliary-set-based entropy inversion from the reported scalars. Every privacy metric is reported per round, because Decaf weakens as models converge and early rounds are where leakage is worst.
Decisive checks. There are four:
- Does sign compression or 8-bit quantization leave Decaf's null-class accuracy at 100%?
- At what Top-k sparsity, if any, does GDBR fall below 50% label recovery under FedAvg with error feedback?
- Does entropy-only selection keep at least half of HiCS-FL's speedup over random sampling on CIFAR-10 Dirichlet partitions?
- Does secure aggregation of the head add under 5% to uplink bytes at 99% body sparsity?
A failure on check 3 would mean class identity is genuinely needed for good selection, and the privacy-speed trade-off is unavoidable. That would itself be worth reporting.
References
- Heterogeneity-Guided Client Sampling: Towards Fast and Efficient Non-IID Federated Learning
- Decaf: Data Distribution Decompose Attack against Federated Learning
- Building Gradient Bridges: Label Leakage from Restricted Gradient Sharing in Federated Learning
- Breaking Secure Aggregation: Label Leakage from Aggregated Gradients in Federated Learning
- Label Leakage in Vertical Federated Learning: A Survey
- Mixed Quantization Enabled Federated Learning to Tackle Gradient Inversion Attacks
- Soteria: Provable Defense Against Privacy Leakage in Federated Learning From Representation Perspective
- Heterogeneity-Aware Federated Learning with Adaptive Client Selection and Gradient Compression
- Towards Eliminating Hard Label Constraints in Gradient Inversion Attacks
- Evaluating Gradient Inversion Attacks and Defenses in Federated Learning