Cross-lingual voice cloning means making a person's voice speak a language they never recorded. These systems face a well-documented tension. Keep the speaker's voice faithfully and you also keep their foreign accent; make the speech sound native and the voice drifts toward a generic one. Recent open benchmarks report this tension as hard numbers. On an English-to-Arabic benchmark, an autoregressive model (one that predicts speech tokens one at a time) reaches word error rate near 0.10 but speaker similarity near 0.51. A diffusion model (a generator that turns noise into speech by repeated denoising) reaches similarity of 0.72–0.76 but makes more pronunciation errors (Toward Speaker-Preserving Arabic Speech). Those similarity numbers come from cosine similarity between speaker-verification embeddings. Nobody has checked how much of that score responds to the voice, meaning vocal-tract timbre, and how much responds to the way of speaking: rhythm, intonation and accent, the very things a native-sounding system is supposed to change.
This proposal builds an open, reproducible way to measure identity across languages that does not reward a system for keeping its accent. We then use it to re-measure the trade-off. The central claim is testable. A speaker-similarity metric will score accented output higher than native-sounding output even when the timbre is identical, and correcting for this will shrink the apparent identity gap between system families. If the gap does not shrink, the trade-off is real and the field gains a validated metric. If it does shrink, published comparisons have been partly measuring accent retention.
Core Claim
Two biases are entangled in current cross-lingual evaluation. Human raters judge similarity poorly when the reference clip is in a different language from the test clip. The Voice Conversion Challenge 2020 found that this mismatch imposes high cognitive load and depresses similarity ratings, with cross-lingual systems staying below 80% "same speaker" against over 90% intra-lingually (Voice Conversion Challenge 2020). The field's answer has been to replace listeners with ECAPA-TDNN cosine similarity (ECAPA-TDNN is a standard speaker-verification network) and Whisper word error rate, as the Arabic benchmark does. That swap assumes the embedding is insensitive to language-specific prosody. Speaker-verification embeddings are trained to separate people, and people differ in rhythm and intonation as well as timbre, so there is good reason to doubt the assumption.
We propose to measure that sensitivity directly. We then report identity on a scale anchored by real human recordings. Its ceiling is how similar the same bilingual person sounds to the encoder across their two languages. Its floor is how similar different people sound across languages. A system score $s$ becomes
$$s_{\text{cal}} = \frac{s - \mu_{\text{diff}}}{\mu_{\text{same}} - \mu_{\text{diff}}},$$
where $\mu_{\text{same}}$ is the mean encoder cosine between natural recordings of one speaker in language A and language B, and $\mu_{\text{diff}}$ is the same quantity for different speakers. A value of 1 means "as similar as the real person speaking the other language." This anchoring is what makes the number comparable across encoders, languages and labs. Without it, a 0.51 versus 0.72 gap has no interpretable meaning.
Research Program
The work has three parts.
The first part builds a ground-truth anchor set from natural human speech, never synthetic audio. This responds directly to the Arabic benchmark's acknowledged weakness that its target-language references are model-generated and can carry vocoder artifacts and model-specific prosody. We draw on two sources. One is EMIME, the multi-speaker English/Finnish/German/Mandarin corpus that VCC 2020 standardized and that the EMIME project built for personalized speech-to-speech translation (Personalising Speech-To-Speech Translation in the EMIME Project). The other is code-switched utterances in CS-FLEURS, where one speaker produces two languages within the same recording session. A necessary first step is to audit which EMIME speakers have recordings in both their native language and English. If that subset is small, CS-FLEURS becomes the primary anchor, and the language pairs we can calibrate are limited to what it covers.
The second part is a controlled probe that holds timbre fixed while varying only accent and prosody. For this we reuse the articulatory codec SPARC (Coding Speech Through Vocal Tract Kinematics). SPARC represents speech as 12 vocal-tract movement traces plus pitch and loudness, and supplies the speaker's timbre as a separate 64-dimensional embedding. It reports coding-recoding similarity of 0.944, so its rendering keeps the voice. Here SPARC serves as a measuring instrument, not a synthesis method. We render the same speaker embedding with three kinds of movement and pitch trajectories: those taken from a native speaker of the target language, those taken from a non-native speaker reading the same text, and hybrids. The hybrids follow the duration-swap idea of HMM based TTS for mixed language text, transplanting native timing while leaving pitch shape alone, and vice versa.
Any change in encoder cosine across these renderings comes from accent, timing or pitch, not timbre. The probe yields a per-factor sensitivity for each candidate identity metric: ECAPA-TDNN, a WavLM-based verifier, and the alternatives below. We will also test whether SPARC renders native and accented trajectories with equal quality. If it does not, rendering quality is confounded with accent. We control for this with an automatic naturalness score and by matching probe pairs on that score.
The third part re-evaluates systems. We score the systems compared in the Arabic benchmark: autoregressive Qwen3-TTS, diffusion VoxCPM2 and flow-matching CosyVoice3, plus Cross-Lingual F5-TTS (Cross-Lingual F5-TTS). We use its Arabic, German, French, Japanese, Russian and Chinese prompt sets, the EMIME language pairs, and FLEURS-based prompts. For each system we report raw similarity, calibrated similarity, the probe-corrected similarity, word error rate, and an accent score from a native-versus-non-native classifier trained on the anchor set. We then ask whether the frontier traced by these systems moves when identity is measured properly. All code, anchor lists and probe audio will be released, because the point is a protocol others can rerun without private data. The proprietary 550-hour corpus behind Learning to Speak Fluently in a Foreign Language is the kind of resource this work deliberately does not need.
Baselines
The naive reference point is raw ECAPA-TDNN cosine against an English reference, exactly as used in Toward Speaker-Preserving Arabic Speech. That paper reports similarity of 0.72–0.76 for VoxCPM2, about 0.51 for Qwen3-TTS and 0.703 for OmniVoice, with Arabic word error rates of about 0.10 for Qwen3-TTS and 0.238 for OmniVoice. We recompute these with the same Whisper large-v3 and ECAPA-TDNN checkpoints on the same released pairs, first confirming that we reproduce the reported values. That confirmation is itself a reproducibility check.
The strongest existing human method is the VCC 2020 cross-lingual similarity test, in which a listener compares a test clip with a reference clip in the other language. VCC 2020 reports best cross-lingual systems below 80% "same speaker" and intra-lingual systems above 90%. We run this test on our EMIME and CS-FLEURS subsets with the same rater pool and number of trials as our gold protocol.
The gold criterion against which every metric is judged is a language-matched human test. Listeners compare a system's output in language B with a natural recording of the same bilingual person in language B. This test removes the cognitive-load bias and is only possible because the anchor speakers are real bilinguals.
A metric beats raw ECAPA if it has higher Spearman rank correlation with the gold test across systems and utterances. It must also shrink the probe sensitivity to accent and timing to within the spread of same-speaker scores. Correlations will be bootstrapped over speakers so that an improvement is not driven by a handful of voices.
The ablation that isolates the key ingredient is calibration without the probe correction, using only the same/different anchoring. This separates "putting scores on an interpretable scale" from "removing prosody sensitivity."
Why not just…?
Why not just run more human similarity tests? VCC 2020 already showed that the cross-language reference itself biases listeners, and that the cost of crowdsourced tests limited it to 25 test sentences. More raters do not remove a systematic bias. Our baseline comparison measures how far the cross-language human test diverges from the language-matched gold test. That gold test is the only human protocol we treat as ground truth, and it cannot scale to arbitrary languages because it needs real bilingual speakers.
Why not just swap in a stronger speaker encoder, or average several? This is a genuine threat. A different encoder might simply be less prosody-sensitive. We do not argue it away; we treat every encoder as a candidate metric and run each through the same probe and gold correlation. If an off-the-shelf encoder already has near-zero accent sensitivity and tracks the gold test, the contribution reduces to the calibrated scale and the probe as a certification tool. That would still change how numbers like 0.51 versus 0.72 should be read.
Why not just flatten the pitch or neutralize prosody before scoring? This is also plausible, and it is cheap. We include it as a candidate metric: both clips are resynthesized through SPARC with pitch set to a fixed contour and timing equalized. Its weakness is predictable from Tone Learning in Low-Resource Bilingual TTS. Pitch carries lexical meaning in Mandarin, and flattening it there may also remove the speaker's habitual pitch range, which is part of identity. The probe will show whether neutralization removes accent sensitivity without collapsing the separation between same and different speakers. If it does both, we adopt it as the protocol's default.
Decisive Checks
The first check concerns the probe. If ECAPA cosine with timbre held fixed differs between native and accented trajectories by less than the natural variation between two same-language utterances of the same speaker, the core hypothesis fails. In that case we report ECAPA as validated for cross-lingual use, which is still a useful result.
The second check is whether the identity gap between Qwen3-TTS and VoxCPM2 narrows under the corrected metric while their word-error-rate ordering is unchanged. Narrowing would mean part of the "autoregressive models wash out identity" finding is accent retention being scored as identity.
The third check is whether the corrected metric's ranking of systems agrees better with the language-matched human test than either raw cosine or the VCC-style cross-language human test does. Together these three outcomes determine whether the nativeness-identity trade-off, as currently reported on open benchmarks, describes the systems or the ruler used to measure them.
References
- Toward Speaker-Preserving Arabic Speech: A Bilingual Resource and Comparative Evaluation of Cross-Lingual Voice Cloning
- Voice Conversion Challenge 2020 -- Intra-lingual semi-parallel and cross-lingual voice conversion --
- Personalising Speech-To-Speech Translation in the EMIME Project
- CS-FLEURS: A Massively Multilingual and Code-Switched Speech Dataset
- Coding Speech Through Vocal Tract Kinematics
- HMM based TTS for mixed language text
- Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis
- Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning
- Tone Learning in Low-Resource Bilingual TTS