Learning the Space of Voices from Locally Privatized Speaker Adaptations

Most neural text-to-speech systems learn from one clean studio voice. That makes them brittle. They mix up what belongs to one person (vocal tract length, habitual pitch, pacing) with what belongs to language, so they cannot produce or adapt to new voices, accents, or speaking styles. The usual fix is to pool audio from thousands of speakers on a server. Voice is biometric data, though, and the populations most missing from public corpora are often the ones least willing or least permitted to upload recordings. This note proposes a different way to get cross-speaker knowledge. Each phone adapts a small, fixed set of about 5,000 voice-specific parameters on its owner's own recordings. It then sends the server one noisy, clipped copy of that parameter vector, protected by local differential privacy: a guarantee that holds even if the server is an adversary, because each device adds calibrated noise before anything leaves it. The server never sees audio or clean parameters. It estimates the population's dominant directions of voice variation and sends back a compact prior. New users can then adapt a good voice from only a handful of sentences.

The idea rests on one statistical fact. Local privacy noise is heavy for any single user but isotropic, meaning it is spread equally in every direction. Isotropic noise inflates every direction of a covariance matrix by the same constant, so it does not bias which directions dominate. The shape of the voice population therefore survives privatization and can be recovered by averaging over many users, even when no individual report is usable. The proposal treats speaker variability as the thing to be learned and local privacy as the constraint. The compact adaptation space of AdaSpeech: Adaptive Text to Speech for Custom Voice serves as the bridge between them.

Core Claim

A population prior over speaker adaptation parameters can be estimated under user-level local differential privacy accurately enough to improve few-shot voice adaptation for speakers unlike those in any public corpus. The prior consists of a mean, a low-rank principal subspace, and residual variances. The privacy cost is paid once per user over their whole adaptation set, so it does not grow with audio length. The user's own synthesized voice is never degraded, because they keep their clean local parameters and only the report sent to the server is noised.

This differs in kind from the closest existing work. Differentially Private Speaker Anonymization adds Laplace noise frame by frame to speech-recognition features and pitch. Its goal is to erase speaker identity for transcription, and its guarantee weakens linearly with utterance length. Here, speaker identity is the signal the population statistic must preserve, and the privacy unit is the whole user. FedSpeech: Federated Text-to-Speech with Continual Learning and Federated Learning with Dynamic Transformer for Text to Speech train synthesis models across devices. They share unnoised weight updates and offer no formal privacy guarantee. The latter was evaluated only on a single-speaker corpus split into five clients. AdaSpeech showed that speaker variability fits in about 5K Conditional Layer Normalization parameters, which are per-speaker scale and bias vectors for the decoder's normalization layers. It never examined these parameters under privacy, and it never pooled them across users.

Research Program

The system starts from a frozen FastSpeech 2 backbone with AdaSpeech-style Conditional Layer Normalization. We will study two base models: one pre-trained on a public multi-speaker corpus, and one pre-trained on a single speaker. The base stays frozen throughout, so every user's parameter vector lives in the same coordinate system.

Each user $u$ adapts a vector $\theta_u \in \mathbb{R}^d$, with $d \approx 5000$, on their own recordings, exactly as AdaSpeech does. The device then applies two steps before reporting. First, it subtracts the current public mean and clips the result to Euclidean norm $C$. Second, it adds Gaussian noise $\xi_u \sim \mathcal{N}(0, \sigma^2 I)$, with $\sigma$ set by the per-user budget $(\epsilon, \delta)$ and by $C$.

The server estimates the covariance of the noisy reports $\tilde\theta_u$ and subtracts the known noise term:

$$\hat\Sigma = \frac{1}{N}\sum_{u=1}^{N} (\tilde\theta_u - \bar\theta)(\tilde\theta_u - \bar\theta)^\top - \sigma^2 I,$$

where $N$ is the number of reporting users and $\bar\theta$ is the mean report. Because the noise term is a multiple of the identity, the top-$k$ eigenvectors of the raw covariance are unbiased estimates of the true speaker subspace. By standard matrix-perturbation arguments, the error in the estimated subspace angle should scale roughly as $\sigma^2\sqrt{d/N}$ divided by the eigengap, the drop between the $k$-th and $(k{+}1)$-th eigenvalues.

This relation drives the design in two ways. It predicts how many users are needed at a given $\epsilon$. It also motivates a second round of reporting. Once a $k$-dimensional subspace with $k \ll d$ is published, users report only their $k$ coordinates. That second round costs additional budget, which we account for by composition. The noise per coordinate is then no longer spread across 5,000 dimensions, which yields a tighter mean and residual variance.

This is the dimension-reduction lesson from Task-aware Privacy Preservation for Multi-dimensional Data and DP-BART for Privatized Text Rewriting under Local Differential Privacy, with one change. The low-dimensional space is learned from the private population itself rather than fixed by a pre-trained encoder or chosen by pruning. The isotropic-noise reasoning is the same one that makes the input Hadamard transform work in Marginal Release Under Local Differential Privacy, transferred from binary marginals to continuous parameter vectors.

A new user with little data adapts by maximum a posteriori estimation:

$$\theta^\star = \arg\min_\theta \; \mathcal{L}_{\text{mel}}(\theta; D_u) + \lambda\,(\theta - \hat\mu)^\top \hat\Sigma_k^{-1} (\theta - \hat\mu).$$

Here $\mathcal{L}_{\text{mel}}$ is the spectrogram reconstruction loss on the user's few sentences $D_u$, $\hat\mu$ is the privately estimated mean, $\hat\Sigma_k$ is the low-rank-plus-diagonal covariance, and $\lambda$ sets the prior strength. AdaSpeech reports that quality "drops off quickly when the adaptation data is fewer than 10 sentences." That is exactly the regime in which a good prior should matter.

We will also test a secondary use: sampling new synthetic voices from the estimated prior for data augmentation. Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention found that single-voice synthetic data makes downstream classifiers overfit, which makes this use worth checking. It is not the core claim.

The analysis side has three parts. First, we will measure the real eigenspectrum of speaker adaptation vectors on non-private data, to check that a clear eigengap exists. If it does not, the idea fails early. Second, we will compare measured subspace error against the predicted scaling in $N$ and $\epsilon$. Third, we will run empirical privacy attacks against the reports. We note one tension raised by FedSpeech. FedSpeech argues that averaging across speakers blurs voices, which is why it isolates parameters. Our design sidesteps the problem: we average statistics about voices, never the voices themselves, and each user's synthesis still uses unaveraged local parameters.

Baselines

The naive reference point is local-only adaptation: AdaSpeech run on each test speaker's few sentences with no population prior. This is the parameter isolation FedSpeech advocates, and it costs no privacy at all. AdaSpeech reports a similarity mean opinion score (SMOS) of 3.96 on VCTK with about 20 sentences. At that data size we expect little headroom. At 1 to 5 sentences we expect local-only adaptation to degrade. Beating it means a statistically significant SMOS gain and higher automatic speaker-verification similarity at 5 or fewer sentences, with no loss in naturalness MOS.

The strongest competitor, and the one a practitioner would actually deploy, is a public-only prior. It uses the same MAP adaptation, but the mean and subspace are estimated from clean adaptation vectors of public-corpus speakers. It costs nothing in privacy, and it is the real threat. It is run with the identical backbone, rank $k$, $\lambda$ search grid, and test speakers. The private method beats it only if, on test speakers drawn from a population the public corpus under-represents, it achieves higher SMOS and verification similarity at matched adaptation data. The file contains no reported result for any locally private multi-speaker TTS system; no reported result found on Lacuna for LDP-trained speaker priors or LDP federated TTS.

The upper reference is the oracle prior: the same estimator applied to clean private vectors. It shows how much of the achievable gain privacy costs at each $\epsilon$.

The strongest existing decentralized method is FedSpeech. It reports MOS 3.77 and speaker similarity 0.8786 on reduced VCTK, against FedAvg at 3.58 and centralized multi-task training at 3.82. These numbers come without any privacy guarantee. We report FedSpeech as a non-private reference. To represent "just privatize federated learning," we add per-client locally noised FedAvg on the decoder, clipped and noised at the same per-user $\epsilon$ and trained with the same total client compute. Matching it would already count as success, since our method sends 5K numbers once rather than millions per round.

Two ablations isolate the key ingredient. The first replaces the learned private subspace with a random or Hadamard-rotated subspace of the same rank $k$. This tests whether estimating the population's directions matters, as opposed to merely reducing dimension. The second privatizes and pools only the global speaker embedding. AdaSpeech's embedding-only tuning reached SMOS 3.34 against 3.96 for Conditional Layer Normalization, which suggests the richer space carries information that the embedding alone misses.

Why not just…?

Why not just keep everything on-device, as FedSpeech does? Isolation protects privacy perfectly but teaches the shared system nothing about the population. New users with very little data are then stuck with a prior shaped by whoever trained the base. The local-only baseline measures this directly. If it matches our method at 1 to 5 sentences, the population prior adds nothing and we would say so.

Why not just build the prior from a public multi-speaker corpus like LibriTTS? This is the most serious objection. If public audiobook readers already span the relevant voice space, private collection is unnecessary. Our answer is empirical and depends on population mismatch. LibriTTS is read, largely native English speech. Single-speaker corpora such as TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese show that for many languages even that much public data does not exist. The experiments therefore vary how badly the public prior matches the test population. We report where, if anywhere, private reports close a gap that public data cannot.

Why not just run differentially private federated averaging on the whole model? With local noise, perturbing millions of decoder parameters per round drowns the signal. DP-BART's unpruned private baselines scored below 0.15 F1 against 0.65 after dimension reduction, and task-agnostic noise in the task-aware LDP study cost up to 70% in error. Repeated rounds also compose the budget. Central differential privacy with secure aggregation would fare better, but it assumes cryptographic infrastructure and a threat model this proposal deliberately does not require. The noised-FedAvg baseline tests the local version directly.

Why not just anonymize the audio locally and upload it? Differentially private anonymization works by replacing the speaker with a pseudo-speaker. That removes exactly the between-speaker variation the server is trying to learn. It also carries frame-level budgets that grow with duration. Its reported strength, speaker identification dropping from 97% to 14% at $\epsilon=1$, is a measure of identity removal. Here that would be a measure of lost utility.

Proposed Experiments

The primary setting uses the AdaSpeech pipeline. The base is pre-trained on a public subset of LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech, a corpus of 585 hours from 2,456 speakers. A disjoint set of roughly 1,000 to 2,000 LibriTTS speakers acts as private clients, each adapting on their own utterances. Test speakers are the 108 accented VCTK speakers, adapted with 1, 3, 5, 10, and 20 sentences.

A harder mismatch setting builds the public prior from a deliberately narrow slice, such as one gender or a handful of speakers, while the private clients span the full population. A third setting starts from a single-speaker LJSpeech base, to test whether private priors help escape the single-speaker bottleneck when the backbone itself has seen one voice.

We sweep per-user $\epsilon \in \{1, 2, 4, 8\}$ at $\delta = 10^{-5}$, the number of clients $N$, and the rank $k$. We run both the one-round protocol and the two-round protocol, with composition accounted for.

We measure naturalness MOS and SMOS through listening tests, cosine similarity from an independent speaker-verification model, and spectral distortion. For privacy, we report the formal per-user $\epsilon$ alongside an empirical attack. The attacker receives a user's noisy report plus a pool of candidate recordings and tries to identify the speaker. We report identification accuracy and equal error rate, set against the anonymization paper's LibriSpeech figures for context.

Three checks decide whether the idea holds. First, the clean eigenspectrum of adaptation vectors must show a clear eigengap within the first few dozen components. Second, the principal angle between private and oracle subspaces should shrink with $N$ as the $\sigma^2\sqrt{d/N}$ argument predicts. A large deviation would mean the clipping bias, which the formula ignores, dominates. Third, and most important, the private prior must beat the public-only prior on SMOS and verification similarity at 5 or fewer sentences in the mismatch settings at $\epsilon \le 4$. It must also beat the random-subspace ablation, so that the gain can be credited to the population structure rather than to dimension reduction. If the public prior is never beaten, the conclusion is that public voice diversity is already sufficient for this adaptation space. That negative result is itself useful for deciding whether voice data needs to be collected privately at all.