Unsupervised Polyglot Text To Speech

Eliya Nachmani, Lior Wolf

Abstract

We present a TTS neural network that is able to produce speech in multiple languages. The proposed network is able to transfer a voice, which was presented as a sample in a source language, into one of several target languages. Training is done without using matching or parallel data, i.e., without samples of the same speaker in multiple languages, making the method much more applicable. The conversion is based on learning a polyglot network that has multiple per-language sub-networks and adding loss terms that preserve the speaker's identity in multiple languages. We evaluate the proposed polyglot neural network for three languages with a total of more than 400 speakers and demonstrate convincing conversion capabilities.

Introduction

In Text-to-Speech (TTS) research, "multilingual" systems can speak several languages, but "polyglot" systems—where a single voice can speak multiple languages—are significantly more challenging to develop. The primary obstacle is the lack of parallel data: recordings of the same individual speaking multiple languages are rare and expensive. Most high-quality speech datasets are monolingual. This research addresses the challenge of transferring a speaker's unique vocal identity into a language they have never spoken, using only unsupervised, monolingual data for training.

Proposed Architecture

The system builds upon the VoiceLoop architecture, a biologically inspired TTS system using a phonological buffer for context. To enable polyglot capabilities, the authors introduced several architectural innovations:

  1. Language-Specific Encoders: Separate encoders for each language map phonemes into a shared, language-independent vector space.
  2. Shared Decoder: The attention mechanism, buffer updates, and output networks are shared across all languages, facilitating cross-lingual transfer.
  3. The $L_{poly}$ Loss: This core innovation preserves speaker identity. During training, the model generates a sample of a speaker (e.g., English) in a target language (e.g., German). The $L_{poly}$ loss forces the speaker embedding of this generated audio to match the original source embedding.
System architecture featuring language-specific encoders and a shared decoder.

Experimental Setup

The model was evaluated using three large-scale datasets:

  • English: VCTK (109 speakers, ~40,000 samples).
  • Spanish: DIMEx100 (100 speakers, ~5,500 samples).
  • German: PhonDat1 (201 speakers, ~15,000 samples).

The evaluation focused on "unseen" speakers to test the model's generalized cloning capabilities.

Results and Identity Disentanglement

A key finding is the successful disentanglement of speaker identity from linguistic features. Visualization of the speaker embedding space reveals that without the $L_{poly}$ loss, embeddings cluster by language. With the loss, they cluster by identity, proving the model separates "how a person sounds" from "the language they speak."

Speaker embedding visualizations: (a) without $L_{poly}$ loss, embeddings cluster by language; (b) with $L_{poly}$ loss, they cluster by identity.

Quantitative results using Mean Opinion Scores (MOS) showed high naturalness (above 3.0) and strong speaker similarity. For instance, an English voice transferred to Spanish achieved a similarity score of 3.41, nearly matching the 3.51 score for English-to-English synthesis. Furthermore, the $L_{poly}$ loss improved neural network-based speaker identification accuracy; for German-to-English transfer, accuracy rose from 82.71% to 92.29%.

Conclusion

This unsupervised approach demonstrates that high-quality polyglot TTS is possible without parallel data. While performance is sensitive to dataset size and limited by current vocoder technology, the ability to perform zero-shot voice transfer across languages opens new possibilities for personalized digital assistants and content localization. Future work will explore more diverse language families and modern neural vocoders.

Summary updated

Abstract

We present a TTS neural network that is able to produce speech in multiple languages. The proposed network is able to transfer a voice, which was presented as a sample in a source language, into one of several target languages. Training is done without using matching or parallel data, i.e., without samples of the same speaker in multiple languages, making the method much more applicable. The conversion is based on learning a polyglot network that has multiple per-language sub-networks and adding loss terms that preserve the speaker's identity in multiple languages. We evaluate the proposed polyglot neural network for three languages with a total of more than 400 speakers and demonstrate convincing conversion capabilities.
BibTeX
@inproceedings{nachmani2019unsupervised,
  author = {Eliya Nachmani and Lior Wolf},
  title = {{Unsupervised Polyglot Text To Speech}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=bGE9h8Ioit},
}
Figure