Encrypting the Voice, Not the Gradient: Identity-Targeted Selective Encryption for Federated Speech Training

Federated learning trains a shared model without collecting users' recordings. Each phone sends only a gradient, the direction in which the model's weights should move. For speech, that gradient is no longer a safe summary. How to Recover Long Audio Sequences Through Gradient Inversion Attack With Dynamic Segment-based Reconstruction reconstructs up to 140 seconds of audio from shared gradients. On Speech Commands V2 it reaches 47.7 dB signal-to-noise ratio, against 32.4 dB for earlier attacks, and the recovered audio is good enough for a speaker-verification system to re-identify the speaker. The cryptographic fix is homomorphic encryption, a way of encrypting numbers so that the server can still add them without decrypting. Encrypting every gradient value is costly: Evaluating Selective Encryption Against Gradient Inversion Attacks cites roughly 100x communication overhead. That paper, and SHE-LoRA, show that encrypting a small, well-chosen slice of the gradient can stop reconstruction of images and text.

This proposal asks a different question for speech. The aim is to stop the attacker from learning who is speaking, and that target is not the same as stopping a good reconstruction. Voice anonymization research shows that identity survives in rhythm, intonation, phone durations and word choice after the voice's timbre has been replaced (Disentangling prosody and timbre embeddings via voice conversion; Private kNN-VC). It also shows that informed attackers who retrain on degraded audio recover much of it (Introducing the VoicePrivacy Initiative). Our bet has two parts. First, a blurry reconstruction can still give away the speaker. Second, the gradient values worth encrypting are the ones that vary between speakers, not the largest ones. If the bet is right, encryption chosen to target identity should protect identity at a far smaller encrypted fraction than encryption chosen by gradient size.

Core Claim

We claim three things. The cheapest encryption mask, meaning the set of gradient coordinates sent encrypted, is one ranked by how much each coordinate separates speakers. Reconstruction-fidelity thresholds of the kind used in prior selective-encryption work give a false sense of safety for voice. And light on-device anonymization combined with an identity-targeted mask reaches near-chance speaker verification at lower total cost than either tool alone.

Concretely, we score gradient coordinate $j$ with a speaker Fisher ratio. The score is computed by the server on public multi-speaker data at the current global weights:

$$s_j = \frac{\mathrm{Var}_{k}\big(\mathbb{E}[g_j \mid \text{speaker}=k]\big)}{\mathbb{E}_{k}\big[\mathrm{Var}(g_j \mid \text{speaker}=k)\big]}$$

Here $g_j$ is coordinate $j$ of a per-utterance gradient and $k$ indexes speakers. The numerator measures how much the average value of that coordinate differs from one speaker to the next. The denominator measures how much it varies among one speaker's own utterances. Clients encrypt the top fraction $\rho$ of coordinates by $s_j$ and send the rest in plaintext.

Because the score uses only public data and public weights, the mask leaks nothing about clients. Every client can use the same mask, which sidesteps the per-client mask negotiation that SHE-LoRA needs for heterogeneous devices. The server recomputes the mask every few rounds as the weights drift. Server-side aggregation needs only homomorphic additions, so it requires no bootstrapping, the costly noise-refresh step that dominates fully homomorphic computation (Circuit Bootstrapping: Faster and Smaller). Cost therefore scales with ciphertext size, which is set by $\rho$.

Research Program

The setting is federated training of two models. The first is a keyword-spotting model on Speech Commands V2, chosen for direct comparability with the Zeng and Rudzicz attack. The second is a small speech recognizer on LibriSpeech train-clean-100, with one speaker per client and evaluation speakers disjoint from the public speakers used to compute $s_j$. Gradients are encrypted under CKKS (Floating-Point Homomorphic Encryption). We record ciphertext bytes per round and client encryption time, packing the encrypted coordinates contiguously so that they fill the parallel slots of each ciphertext, as SHE-LoRA's column-swapping trick does.

The attacker is honest-but-curious: it follows the protocol but inspects everything it receives. It runs at three strengths, and the strongest one is the main result.

The first is the dynamic segment-based inversion of Zeng and Rudzicz. We adapt it so the gradient-matching loss runs only over the plaintext coordinates, then convert the reconstruction to audio and score it with a pretrained ECAPA-TDNN speaker verifier, the same verifier that paper used.

The second is the VoicePrivacy "informed" attacker. It retrains the verifier on reconstructions produced the same way, so it learns to read identity through the reconstruction artifacts.

The third skips reconstruction altogether. A network maps the plaintext gradient coordinates directly to a speaker embedding. It is trained on gradients from public speakers and tested on held-out speakers. This attacker matters most for our claim, because it targets identity directly and cannot be fooled by poor reconstructions.

We also run a text-only verifier on Whisper transcripts of the reconstructions. It follows You Are What You Say, which shows that transcripts alone identify speakers. This attacker measures the linguistic identity channel that no acoustic defense touches.

The input conditions are raw audio, audio passed through the VoicePrivacy x-vector baseline, and audio passed through the differentially private anonymizer of Differentially Private Speaker Anonymization. Each condition is crossed with client batch sizes 1, 8 and 32. Batch size matters because Concealing Sensitive Samples against Gradient Leakage in Federated Learning shows that per-sample gradients that are nearly unrelated to each other can be pulled apart from a batch average.

The primary privacy metric is the equal error rate (EER) of the strongest attacker. EER is the operating point where false accepts equal false rejects, and 50% is chance. Secondary privacy metrics are reconstruction SNR and cosine similarity, which let us show where these disagree with EER. Utility metrics are keyword accuracy and word error rate (WER).

Baselines

The naive reference is plaintext federated training on raw audio with no defense. It shows how much identity the attack recovers when nothing stops it. For Speech Commands V2 the closest published figure is the 47.7 dB SNR and 0.98 average cosine similarity reported by Zeng and Rudzicz. They report no EER figure on LibriSpeech, so we measure that ourselves.

The upper reference is full homomorphic encryption of every coordinate, which gives perfect protection at maximal cost. It sets the byte and latency scale against which selective schemes are judged. We found no reported result on Lacuna for full homomorphic encryption cost in federated speech training.

The strongest existing method is magnitude-based selective encryption ("Grad") from Gu et al. That work reports that encrypting 10% to 40% of gradients reaches LPIPS ≥ 0.3 on CIFAR-100 and ROUGE-1 ≤ 0.2 on text. LPIPS measures perceptual image difference, and ROUGE-1 measures word overlap with the original text. Grad adds no computation and is the method that beat optimization attacks in that study, which makes it the natural thing a practitioner would port to speech. We will also run their parameter-magnitude ("Param") and gradient-times-weight ("ProdSig") scores.

SHE-LoRA's Wanda-based importance mask is a second strong competitor. SHE-LoRA reports defeating DAGER, an analytical attack that solves for inputs directly rather than by search, at 0.03% encrypted parameters. We run it on a LoRA-adapted version of the recognizer. We found no reported result on Lacuna for any selective-encryption scheme on speech.

Differentially private noise is the perturbation alternative. It enters in two places: as DP-SGD (per-sample clipping plus noise) on the client gradients, and as the anonymizer of Shamsabadi et al. on the inputs. For the anonymizer, Shamsabadi et al. report EER rising from 15% to 30% and WER from 5% to 7% at $\epsilon=1$ on LibriSpeech test_clean.

All methods share the same data splits, model and number of federated rounds. Each gets the same budget for tuning its threshold or noise scale, and comparisons are made at matched encrypted fraction $\rho \in \{0.1\%, 1\%, 5\%, 10\%\}$ or at matched bytes per round.

Two ablations isolate the key ingredient. In the first, a "content Fisher ratio" mask ranks coordinates by between-transcript rather than between-speaker variance. It tests whether speaker specificity, rather than any data-dependent ranking, drives the gain. In the second, a mask frozen at initialization tests whether recomputing the mask over rounds matters.

Beating Grad means the following. Under the strongest attacker, the identity-targeted mask reaches EER ≥ 45% at no more than half the encrypted fraction Grad needs, on both tasks and at batch size 1, with WER and keyword accuracy within noise of plaintext training.

Why not just…?

Why not just port Grad-based selective encryption with a reconstruction-quality threshold? This is the real threat. If gradient size and speaker separability pick out largely the same coordinates in speech models, Grad will match our mask and costs nothing to compute. The experiment settles it directly in two ways: we compare the overlap of the two masks, and we compare EER at matched $\rho$. We also check whether Grad's reconstruction-fidelity threshold is safe for identity by locating operating points where SNR has collapsed but the gradient-to-embedding attacker still verifies speakers well below chance. If Grad matches our mask, the contribution shrinks to that measurement finding, and we would report it as such.

Why not just anonymize the audio on the device and skip encryption? Anonymized speech still leaks identity to informed attackers. VoicePrivacy reports EER falling to 26% to 37% once attackers retrain on anonymized data, and prosody and duration cues survive timbre replacement. The anonymized-input, no-encryption arm measures exactly how much of that residue reaches the gradient. Our hypothesis is that anonymization lowers the encryption budget but does not remove the need for it.

Why not just add differential-privacy noise to gradients? Heuristic noise and pruning can be bypassed by attacks that use generative priors (Auditing Privacy Defenses in Federated Learning via Generative Gradient Leakage). Frame-level privacy guarantees also weaken as utterances get longer, because the privacy loss of each frame adds up. The DP-SGD arm shows what WER cost buys equal EER, compared with the byte cost of encryption.

Why not just use larger client batches? Averaging over a batch makes the inversion problem under-determined, meaning many inputs fit the same gradient (Gradient Inversion with Generative Image Prior). But nearly unrelated per-sample gradients remain separable, and on-device speech clients often hold few utterances per round. The batch-size sweep tests whether batching alone reaches chance EER.

Decisive Checks

The idea stands or falls on three results. First, at matched encrypted fraction, the speaker-Fisher mask must yield higher strongest-attacker EER than Grad, and higher than the content-ratio ablation. Second, we must find cases where reconstruction SNR looks safe while EER does not, which would show that fidelity thresholds mislead for voice. Third, anonymized input plus a small identity mask must reach EER near 50% at fewer bytes per round than raw input with Grad.

Text-only verification on transcripts is reported separately as a ceiling that acoustic encryption cannot lower. If it stays far below chance, the conclusion is that federated speech recognition must also protect the content channel, and we will state that limit plainly.