Separating Voice from Accent Across Clients in Federated Cross-Lingual Speech Synthesis

Suppose we want a phone to speak a language its owner does not speak, in the owner's own voice, without the owner's recordings ever leaving the device. Federated learning, where devices train a shared model locally and send back only parameter updates, handles the privacy side. It creates a specific problem for cross-lingual voice cloning, though. Each device holds speech in one language only. From a single client's data, the speaker's voice and the speaker's language are perfectly confounded: everything the person says carries the same phonology, rhythm and accent. A personal voice adapter trained on that device will absorb the accent along with the timbre. When it is used to speak another language, the result is the familiar "phonetic leakage" of cross-lingual adaptation. Centralized systems avoid this because they see many speakers in many languages and can learn what is voice and what is language. Federated learning hides exactly that cross-speaker view.

The core claim of this proposal is that the cross-client information needed to separate voice from language can be recovered under federated constraints without exposing any individual's adapter. Clients that share a language also share a language-specific component in their personal adapters. A server running secure aggregation (a protocol that reveals only sums of client updates, never individual ones) can estimate that shared component per language. It can then publish a small "accent subspace" that every client projects out of its private voice adapter. At synthesis time in a foreign language, the client keeps the projected voice adapter and swaps in the target language's shared component. We call this federated language centering. It is a federated analogue of the "average voice" idea in classical speaker adaptation, and we expect it to reduce accent leakage without lowering speaker similarity.

Where This Sits Relative to Prior Work

Our setting is defined by the closest federated speech work. Fed-PISA splits a low-rank adapter (LoRA, a small trainable add-on to frozen weights) into a private identity part kept on the device and a shared style part aggregated across clients. It reports speaker similarity 0.645 and naturalness MOS 4.08 on emotional English corpora. FedSpeech isolates per-speaker weights with pruning masks and a private speaker lookup table, and beats FedAvg on VCTK (MOS 3.77 vs 3.58). FedDT-TTS targets communication cost on single-speaker LJSpeech. All three are monolingual. Their private components are therefore never tested on the confound we target: within one client, a private identity adapter has no signal telling it to leave the accent out.

The proposal reuses Fed-PISA's private/shared adapter split as its method base. Its new ingredient is a cross-client constraint on the private part, computed only from aggregated statistics.

The text-only work Breaking Physical and Linguistic Borders motivates the linguistic-distance angle. It shows that federated prompt averaging across languages helps low-resource clients, and that the benefit is larger for languages far from the backbone's representations. Whether this holds for acoustics, where averaging is known to flatten voices, is open.

On the cross-lingual side, the VCC 2020 overview documents the gap we aim to close. Fewer than 80% of cross-lingual samples were judged "same speaker", against over 90% within a language, and Mandarin targets fared worse than German ones. GlowVC acts as a constraint: it reports that pushing speaker similarity higher (AutoVC) costs intelligibility, so leakage and similarity must be measured together. Speaker-independent content representations such as phonetic posteriorgrams (NUS & NWPU, CASIA) and self-supervised discrete units (ZMM-TTS) clean the input side of synthesis. They do not stop a personal adapter on the output side from re-learning the accent, and that output side is the failure our method addresses.

Core Mechanism

Each client $i$ speaks language $\ell(i)$ and holds a few minutes of speech. It adapts a frozen multilingual base model with an update $\Delta_i = A_{\ell(i)} + S_i$. Here $A_\ell$ is a language adapter shared and averaged across all clients of language $\ell$, and $S_i$ is a private speaker adapter that never leaves the device in the clear.

Every few rounds, clients submit their flattened $S_i$ to secure aggregation grouped by language. The server then learns only the per-language means $m_\ell$ and the global mean $\bar m$. The vectors $m_\ell - \bar m$ span an accent subspace with orthonormal basis $U$. Since there are only as many directions as languages, $U$ has very few columns and is cheap to broadcast. Clients then update

$$ S_i \leftarrow (I - UU^\top)\,S_i, \qquad A_\ell \leftarrow A_\ell + U U^\top m_\ell , $$

which moves the language-shared part of every private adapter into the language adapter, where it belongs. During local training, a penalty $\lambda\|U^\top S_i\|^2$ keeps the private adapter from drifting back into the subspace. Here $\lambda$ is a weight tuned on held-out clients.

To speak language $\ell'$, the device combines $A_{\ell'}$ with its own centered $S_i$. The language mean is estimated from $n_\ell$ clients, so its noise should shrink roughly as $1/\sqrt{n_\ell}$ (standard averaging of independent client deviations). We therefore expect leakage reduction to improve with the number of clients per language and to be unreliable for very small groups. Secure aggregation also only hides individuals when groups are not tiny, so we enforce a minimum group size and optionally add differential-privacy noise to the per-language sums. The prompt-tuning paper flags this kind of leakage as a real risk (Breaking Physical and Linguistic Borders).

Research Program

The base model is a multilingual multi-speaker synthesizer conditioned on self-supervised discrete units in the style of ZMM-TTS, pretrained only on public Multilingual LibriSpeech. MLS speakers held out from pretraining serve as additional simulated clients, which lets us vary the number of clients per language from a handful to over a hundred.

The evaluation clients come from two sources. The first is the SIWIS database, which has 36 accentless bilingual or trilingual speakers across English, French, German and Italian. Each SIWIS client is given only one of its languages for adaptation. Its recordings in the other language, never seen in training, become a ground-truth reference for what that person actually sounds like in the second language. Few cross-lingual benchmarks provide this. The second source is the EMIME-based VCC 2020 cross-lingual task: 70 training sentences per Finnish, German or Mandarin target speaker, with English output required. This adds a distant language and allows comparison with a documented listening protocol.

Beyond the method itself, the program asks three analytical questions. How much accent does a private adapter absorb as a function of local data size? Does centering's benefit grow with linguistic distance (Mandarin over German) in the way Breaking Physical and Linguistic Borders found for text? How many clients per language are needed before the estimated subspace helps rather than hurts?

Baselines

The naive reference point is zero-shot cloning with the frozen public base model: the client's reference clips go in as a prompt and no adaptation occurs, which is the regime Voicebox and ZMM-TTS represent. It uses the same base checkpoint and the same reference audio as every other system. To beat it, centering must give higher speaker similarity to the held-out real second-language recordings without worse word error rate.

The local-only baseline adapts on the device with no communication. We run two variants under the same per-client compute budget and tune both on the same held-out clients: a private LoRA, and speaker-embedding search in the style of ASRRL-TTS. The latter reports that embedding adaptation beats fine-tuning below roughly 20 sentences.

Full-model FedAvg represents the known averaging failure; FedSpeech reports MOS 3.58 for FedAvg on VCTK. FedSpeech itself, with its pruning masks and private speaker table, is run on the same clients as the strongest parameter-isolation method.

The strongest existing federated competitor is a Fed-PISA-style split with a private identity LoRA and shared adapters aggregated per language. It uses the same adapter ranks, rounds and communication budget as our method. Its reported 0.645 speaker similarity and 4.08 nMOS (Fed-PISA) are monolingual emotional-speech figures. They give context only and are not a target on our data. For SIWIS cross-lingual synthesis there is no reported result found on Lacuna, so all federated numbers will be produced here.

The key ablation is exactly this Fed-PISA-style split with the projection removed. Our method differs from it in one ingredient only, so any gap isolates centering. A second ablation computes the language means centrally from all clients' adapters, without secure aggregation, to measure what the privacy protocol and differential-privacy noise cost.

As a centralized upper reference, we pool all client audio and train with the same adapters plus a language-adversarial loss. As external context we cite VCC 2020 cross-lingual results: CASIA reports 69% speaker similarity and MOS 3.99, and NUS & NWPU reports naturalness MOS 4.08 and similarity 2.69. These are conversion systems rated by a different listener pool, so we do not claim to beat them directly. We replicate the VCC similarity protocol for our own systems.

Centering counts as a success if, against the uncentered split, it lowers first-language identification on second-language output and raises cross-lingual speaker similarity. It must also keep within-language similarity and word error rate within noise.

Why not just…?

Why not just use a large centralized multilingual model zero-shot and skip federated training? This is the most serious threat. It is the first baseline because it may well win when client languages are well covered by public data. ASRRL-TTS reports that adaptation improves single-sentence similarity by up to 30% over unadapted embeddings, which suggests prompts miss idiosyncratic timbre. That evidence is monolingual, however. The SIWIS bilingual references settle the question directly. If zero-shot matches centered federated adaptation on second-language similarity, the federated component is not justified for these languages, and we will report that. We also test a language the public base saw little of (Finnish via EMIME), where we expect adaptation to matter more.

Why not just adapt locally and never communicate? Local adaptation cannot remove accent, because a single client has no contrast between voice and language. Centering needs other clients by construction. The local-only baselines measure this directly through first-language identification on second-language output. In text, Breaking Physical and Linguistic Borders found local tuning unstable at 30 samples while the federated version held, but we treat that only as motivation.

Why not just use Fed-PISA-style private/shared adapters with per-language aggregation? The private adapter is precisely where accent hides, and nothing in that design pushes it out. This alternative is our main ablation. If it shows no measurable accent leakage, our premise is wrong and centering is unnecessary. The leakage metric on SIWIS answers this before any listening test is run.

Why not just feed the synthesizer speaker-independent units or posteriorgrams? Those representations clean the content input, as in NUS & NWPU and ZMM-TTS. Our base already uses discrete units, so all baselines inherit that benefit, and any remaining leakage comes from the adapted decoder.

Decisive Experiments

The primary test runs on SIWIS. Each client is adapted on one language, and we synthesize the speaker's other language. We measure three things: speaker-verification cosine similarity to that person's real recordings in that language, word error rate from a multilingual recognizer, and the accuracy of a classifier trained to detect the speaker's adaptation language from synthesized speech. The last is our leakage measure. The hypothesis is that centering cuts first-language identification substantially relative to the uncentered split, while second-language similarity to the real recordings rises or holds, avoiding the similarity–intelligibility trade-off GlowVC observed.

The second test uses VCC 2020 cross-lingual targets with the challenge's MOS and same/different speaker protocol. The prediction is that the gain from centering is larger for Mandarin than for German targets, mirroring the distance effect in the VCC 2020 overview.

The third test sweeps clients per language using held-out MLS speakers. It checks the predicted $1/\sqrt{n_\ell}$ improvement in subspace estimates and finds the minimum group size at which centering helps. Communication cost per round is reported for all federated systems.

The idea fails if any of three things happens: the uncentered split shows no leakage, zero-shot prompting matches it, or centering only helps when computed centrally without privacy protection.