In federated learning, many devices train a shared model without sending their raw data to a central server. When the devices hold very different data, one shared model serves most of them poorly. A common fix is clustered federated learning: the server groups clients with similar data and trains one model per group. To decide the groups without seeing the data, recent methods have clients send a "fingerprint" of their data, most successfully the weights of the model's last layer, the linear classifier that turns features into class scores (FedClust). That same last layer is where gradient-based attacks recover labels and inputs most easily. These attacks reconstruct private training data from the model updates a client sends (SPEAR, Gradient Bridges). Clustering methods currently send the most revealing part of the model and call it privacy-preserving.
We propose a direct fix. The server only needs to learn which group a client belongs to, which is a few bits of information, yet clients send thousands of classifier coordinates. We have each client compress its classifier update with a shared random projection down to a few dozen numbers, clip that short vector, and add calibrated Gaussian noise before sending it. The noise gives the release a formal local differential privacy guarantee, meaning each client's report is statistically protected on its own, without trusting the server. Because the noise is spread over a few dozen dimensions rather than thousands, the same privacy budget leaves the group structure far more visible. The goal is private clustering at ordinary mini-batch sizes, together with an attack audit of what the compressed report still leaks.
Core Claim
Private client clustering is a low-dimensional estimation problem, and current methods pay a dimensionality tax for treating it as a high-dimensional one. Under the Gaussian mechanism, a client that releases a clipped $k$-dimensional vector with clipping radius $C$ and noise multiplier $\sigma$ adds noise of expected norm about $\sigma C\sqrt{k}$. Here $\sigma$ is set by the privacy budget $(\epsilon,\delta)$ alone and does not depend on $k$. Noisy pairwise distances between clients therefore blur at a rate proportional to $\sqrt{k}$. A random projection with entries drawn from $\mathcal{N}(0,1/k)$ preserves distances between cluster centers up to a relative distortion of order $1/\sqrt{k}$, which is the Johnson–Lindenstrauss rate. For a fixed $\epsilon$, cluster separability should therefore peak at an intermediate sketch size $k^*$, far below the thousands of coordinates in a classifier layer. Around that optimum it should beat both noising the full classifier and heavier compression.
The empirical motivation comes from CFL-GP, which clusters gradient profiles with SVD and spectral $K$-means. Its authors report near-perfect clustering even when gradients are compressed 10,000-fold, which is direct evidence that cluster identity lives in a very low-dimensional subspace. CFL-GP never adds privacy noise. We turn its compression result into a privacy gain.
The claim is new relative to the closest work in three ways. First, R-DPCFL shows that differential privacy noise smears client updates and breaks clustering. It recovers the clusters by having every client train on its full local dataset in round one, because larger batches shrink the noise, and then applies soft Gaussian-mixture assignment. R-DPCFL shrinks the noise through batch size. We shrink it through the dimension of the report, which leaves normal mini-batch training intact. Second, R-DPCFL is never tested against inversion attacks. Third, FedClust and Clustered FedStack send raw last-layer weights with no formal guarantee and no attack evaluation.
Research Program
The work has three parts.
Leakage audit. Before proposing a defense, we measure what the current proxy leaks. FedClust sends last-layer weights after one local epoch. That is a multi-step weight change, not the single-step gradient that SPEAR is derived for. The last layer is also followed by a softmax rather than a ReLU. It is therefore an open question, not a given, whether SPEAR-style exact inversion applies. We will run three attacks on FedClust's release on CIFAR-10 and SVHN, varying batch size and number of local steps:
- SPEAR adapted to the classifier layer, where the weight gradient is a sum of outer products between per-sample error vectors and penultimate-layer features, so it has rank at most the batch size;
- GDBR-style recovery of the per-class label counts;
- a learned attacker trained on simulated "shadow" clients that predicts a client's label proportions from its release.
If the audit finds little leakage in practice, that is itself a publishable result, and the defense reduces to a cost-free precaution.
Sketch-then-privatize proxy. In round one, each client trains locally with ordinary mini-batches for one epoch, as in FedClust. It forms its classifier update $\Delta W_L$, projects it with a public random matrix $R \in \mathbb{R}^{k \times d}$ seeded by the server, clips the result to norm $C$, and adds $\mathcal{N}(0,\sigma^2C^2 I_k)$. The server clusters the noisy sketches. We will compare three clustering procedures: FedClust's agglomerative clustering, spectral $K$-means from CFL-GP, and a Gaussian mixture as in R-DPCFL.
Theory. The analysis has two goals. The first is to derive $k^*$ as a function of $\epsilon$, the number of clients per cluster, and the separation between cluster centers. The main derivation will balance the projection distortion against the noise that grows with $\sqrt{k}$. The second is to state the privacy guarantee plainly. The client-level guarantee covers the sketch release. A $k$-dimensional sketch with $k$ below the input dimension also rules out exact reconstruction of a single image on information-counting grounds, although partial feature leakage is still possible and is measured empirically.
Residual label leakage. Clustering under label skew must reveal something about each client's label distribution, because that distribution is what defines the groups. We will measure how much of it attackers still recover as $\epsilon$ tightens. The protection we aim for is recovery no better than an attacker who is told only the client's cluster membership.
Baselines
The do-nothing reference points are FedAvg with no clustering and non-private FedClust. FedClust reports 60.25% accuracy versus 38.25% for FedAvg on CIFAR-10 with Dirichlet α = 0.1 (FedClust). These bracket the utility we are trying to keep. We will rerun both in our own code on identical partitions rather than copy the numbers.
The naive private baseline adds Gaussian noise to the full classifier-layer update at the same $(\epsilon,\delta)$, followed by FedClust's clustering. The research file argues this should collapse the clustering, but no measured result is given. We will measure it.
The strongest existing method is R-DPCFL (Differentially Private Clustered Federated Learning): a full-batch first round, soft assignment with a Gaussian mixture, and private model selection with the exponential mechanism. No reported result was found on Lacuna for R-DPCFL's accuracy on FMNIST or CIFAR-10. We will therefore run it from its description on our partitions, with equal privacy budgets and communication rounds and the same hyperparameter search budget of about 20 configurations per method. We will run it in two settings: its native full-batch round, and a memory-capped version with batch size 64 that reflects edge devices.
The non-private upper bounds on clustering speed are CFL-GP and CLoVE. CFL-GP reports an adjusted Rand index of 1.0 within 20 rounds on Rotated MNIST. The adjusted Rand index (ARI) measures agreement with the true grouping, and 1.0 is a perfect match. CLoVE, which clusters vectors of each client's losses on $K$ broadcast models, reports ARI 1.0 within 2 rounds on FMNIST.
The key ablation keeps everything fixed except $k$. It sweeps from $k = d$, which is just the naive private baseline, down to $k = 4$, at several values of $\epsilon$. We will also compare a random projection against a projection learned from public data, to test whether the random matrix is the limiting factor.
We beat R-DPCFL if we match its final personalized accuracy within 1 point at batch size 64, where it must run its capped variant, or exceed it at equal $\epsilon$, with ARI at least as high and attack success no higher.
Why not just…?
Why not just release a noisy label histogram? Each client could send a differentially private count of its labels: a $C$-dimensional vector with low sensitivity. Under pure label skew this is a real threat and may match our method, since the histogram is the quantity that defines the clusters. We include it as a baseline and expect it to be competitive on the Dirichlet-partitioned CIFAR-10 and SVHN splits. It carries no information, however, when every client has the same labels but the inputs differ. Examples are Rotated MNIST, as used in CFL-GP, and writer-based FEMNIST. There the classifier update still separates clients, because rotated inputs produce different features and different classifier updates. We will report both regimes honestly. If the histogram matches or beats us everywhere, the proposal narrows to feature-shift clustering.
Why not just add differential privacy to CLoVE's loss vectors? Loss vectors are already $K$-dimensional and bounded after clipping, so they are naturally suited to privacy noise. CLoVE's authors explicitly leave privacy open. R-DPCFL's third stage is close to this idea, because clients privately pick the model with lowest loss. CLoVE depends on $K$ broadcast models having diverged, which takes rounds, whereas a classifier sketch is available after one epoch. We will run private CLoVE at the same $\epsilon$ as a baseline. If it clusters equally fast, the contribution shifts to the audit and to the analysis of $k^*$, which applies to both kinds of proxy.
Why not just use secure aggregation? Secure aggregation lets the server see only the sum of client updates. The clustering methods here all need per-client distances (FedClust, CFL-GP), and targeted extraction attacks remain possible even through large aggregates (Panning for Gold).
Why not just withhold the classifier layer? Two results rule this out. GDBR recovers labels with over 80% accuracy from the earlier layers alone. FedClust's own analysis of how similar each layer is across clients shows that the early layers carry little cluster signal.
Proposed Experiments
We will run on CIFAR-10 and SVHN with Dirichlet α ∈ {0.1, 0.5} and pathological label skew, on FMNIST, on Rotated MNIST using the 8 rotation groups in the CFL-GP code, and on FEMNIST. The settings use 50 to 200 clients, $\epsilon \in \{1, 4, 8\}$ at the client level with $\delta = 10^{-5}$, and batch sizes {32, 64}.
We will measure the following:
- clustering quality (ARI) after round one;
- the rounds needed to reach ARI 1.0;
- final per-client accuracy, reporting the worst decile of clients separately, because noise hurts minority distributions most;
- attack success: PSNR for SPEAR, label-count accuracy for GDBR, and the learned attacker's error on label proportions;
- the privacy budget spent;
- upload size.
For reference on what counts as meaningful protection, Huang et al. report 86.10% accuracy with a reconstruction dissimilarity score (LPIPS, higher meaning the reconstruction looks less like the original) of 0.73, versus 0.45 with no defense, on CIFAR-10.
Three checks are decisive. First, ARI as a function of $k$ at fixed $\epsilon$ must show the predicted interior optimum; if it rises monotonically as $k$ grows, the theory is wrong. Second, at batch size 64 the sketch must match or beat capped R-DPCFL in accuracy. Third, SPEAR and GDBR run on the sketch must fall to the level of the "cluster membership only" attacker, while the same attacks on raw FedClust releases succeed.
References
- FedClust: Optimizing Federated Learning on Non-IID Data Through Weight-Driven Client Clustering
- SPEAR: Exact Gradient Inversion of Batches in Federated Learning
- Building Gradient Bridges: Label Leakage from Restricted Gradient Sharing in Federated Learning
- Clustered Federated Learning via Gradient-based Partitioning
- Differentially Private Clustered Federated Learning
- Clustered FedStack: Intermediate Global Models with Bayesian Information Criterion
- CLoVE: Personalized Federated Learning through Clustering of Loss Vector Embeddings
- Panning for Gold in Federated Learning: Targeted Text Extraction under Arbitrarily Large-Scale Aggregation
- Evaluating Gradient Inversion Attacks and Defenses in Federated Learning