TL;DR
Researchers have developed a method to synthesize entirely new, "nonexistent" human voices that sit precisely between specific categories—such as a gender-neutral voice or a voice with a specific level of foreign accent. By applying Optimal Transport theory to interpolate between Gaussian Mixture Models (GMMs) in a speaker embedding space, the system can generate high-quality, natural-sounding speech with fine-grained control over speaker attributes.
What Problem Does This Paper Address?
While modern Text-to-Speech (TTS) systems are incredibly realistic, they usually mimic a specific set of recorded speakers. "Speaker generation" is a newer task that aims to create completely new voice identities. A previous state-of-the-art method, TacoSpawn, uses GMMs to model the distribution of speaker characteristics (embeddings) based on categorical metadata like "male" or "female."
However, TacoSpawn struggles with the "in-between." If you want a voice that is exactly halfway between two genders or has a specific degree of language fluency, simply averaging the model parameters often fails because the underlying distributions don't align linearly. This paper addresses the challenge of mid-attribute speaker generation: creating voices with non-categorical, continuous traits without sacrificing the naturalness of the speech.
Methods
The authors build upon the TacoSpawn framework, which consists of a TTS model and a speaker attribute encoder. Their primary innovation lies in how they handle the speaker embedding distributions:
- GMM Representation: They represent different speaker attributes (e.g., Japanese Male, English Female) as distinct GMMs in a 256-dimensional embedding space.
- Optimal-Transport-Based Interpolation: To find a "mid-attribute" voice, they don't just average the means and variances of the GMMs. Instead, they use Optimal Transport (OT) theory to calculate the weighted barycenter of multiple GMMs.
- Wasserstein Distance: They use the $W_2$ (Wasserstein) distance as a cost function. This allows them to "morph" one distribution into another along the most efficient path, ensuring the resulting "mid-attribute" GMM maintains a coherent shape.
- Controllability: By adjusting interpolation weights ($\lambda$), they can move a synthetic voice along a continuous scalar axis (e.g., from 100% native-sounding to 100% foreign-accented).
Datasets
The study utilized three main datasets:
- JVS (Japanese Versatile Speech): 100 Japanese speakers (49 male, 51 female) used to define "native" attributes.
- VCTK: 108 English speakers (47 male, 61 female). In this experiment, these were treated as "non-native" speakers to train the model on foreign-accented Japanese.
- JSUT: A single-speaker Japanese corpus used for pre-training the base TTS model.
- Evaluation Data: 225 nonexistent speakers were generated for subjective testing using the ITA corpus (recitation sentences).
Key Insights and Ideas
- The "Indefinite Order" Problem: A major reason simple GMM averaging fails is that the "order" of mixtures in a GMM is arbitrary. Mixture #1 in the "Male" GMM might correspond to Mixture #3 in the "Female" GMM. Optimal Transport solves this by mathematically finding the best alignment between these components.
- Nativeness as a Spectrum: By treating English speakers as "non-native" Japanese speakers and using a native-language classifier during training, the authors successfully turned a categorical trait (Language) into a controllable spectrum (Fluency).
- Preserving Naturalness: One of the most significant findings is that "mid-attribute" voices don't fall into an "uncanny valley." The interpolated voices were often rated as more natural than the categorical endpoints.
Key Results
The researchers conducted subjective evaluations with 750 listeners using a 5-point Mean Opinion Score (MOS):
- Attribute Control: Kernel Density Estimation (KDE) plots confirmed that "Mid" attribute voices were perceived by listeners as being successfully distributed between the target categories (e.g., perceived as gender-neutral).
- Naturalness: The MOS for mid-attribute speakers was comparable to, or even higher than, the categorical speakers. For example, "female, mid-nativeness" scored 3.41, while "female, native" scored 3.07.
- Cross-Attribute Control: The method successfully handled two-axis control (Gender and Nativeness simultaneously), allowing for the generation of a "gender-neutral, semi-native" speaker.
Limitations and Future Work
- Data Bias: The authors noted that naturalness tended to increase with "nativeness." This is likely because the TTS model was trained primarily on native Japanese data (JSUT and JVS) and never saw ground-truth "non-native" Japanese speech, only English speech used as a proxy.
- Attribute Scope: This study focused on only two axes: gender and language fluency.
- Future Directions: The team plans to expand the diversity of languages and explore other speaker attributes (like age or emotional expression) to further broaden the range of synthesizable identities.
Summary updated