Turning a Monolingual Speaker into Multilingual for a Mixed-language TTS

Ji He, Yao Qian, Frank K. Soong, Sheng ZhaoInterspeech 2012 · OpenReview

Introduction

In modern applications such as car navigation, Text-to-Speech (TTS) systems frequently need to handle mixed-language content—for example, providing English instructions while pronouncing Chinese street names. To maintain a natural user experience, it is critical that the voice identity remains consistent across both languages. However, hiring professional voice talent fluent in multiple languages is both difficult and costly.

This research addresses the challenge of creating a high-quality, bilingual TTS system using recordings from a speaker who only speaks one language. By "rendering" the monolingual speaker's voice into a target language using a reference speaker, the authors bridge the gap between non-tonal languages (like English) and tonal languages (like Chinese).

System Architecture and Methodology

The proposed pipeline transforms a source speaker's voice into a target language through a multi-stage process designed to preserve identity while adopting new linguistic characteristics.

System overview of the proposed mixed-language TTS framework.

1. Speaker Equalization

To minimize the vocal tract differences between the English source speaker and the Chinese reference speaker, the system employs bilinear frequency warping. This ensures that the spectral characteristics of the source speaker are adjusted to match the "space" of the target language's reference voice.

2. Prosody and Pitch Adjustment

Chinese is a tonal language requiring a significantly wider pitch (F0) range than English. The researchers introduced a scaling factor ($\gamma$) to expand the source speaker's pitch dynamics. Additionally, Time Scale Modification (TSM) was used to synchronize speaking rates between the two speakers, ensuring the timing of the rendered speech feels natural.

3. Trajectory Tiling

This is the core rendering engine. The system takes the warped speech trajectories of the Chinese reference speaker and identifies the best-matching "tiles" (5ms frames) from the English speaker’s database.

The trajectory tiling process using a 'sausage' lattice for waveform concatenation.

As shown in the figure above, the system constructs a "sausage-like" lattice of potential frames. A Viterbi-style search is then used to find the optimal path that minimizes spectral distortion while ensuring smooth concatenations between frames. This process creates a new "rendered" Chinese dataset that carries the English speaker's vocal identity.

4. HMM-based TTS Training

The final step involves training a Hidden Markov Model (HMM) TTS system. This model is trained on a hybrid dataset consisting of the speaker's original English recordings and the newly synthesized Chinese "recordings."

Experimental Setup and Results

The study utilized 4.2 hours of speech from a U.S. English female speaker and 3.8 hours from a Mandarin Chinese female speaker. General F0 distributions were estimated using the TIMIT (English) and 863wave (Chinese) datasets.

Key Findings:

  • Subjective Preference: In AB preference tests, human listeners preferred the proposed tiling method over standard frame-mapping baselines 67.5% of the time.
  • Data Efficiency: The researchers observed a law of diminishing returns regarding data volume; approximately two hours of source speaker data was found to be sufficient for high-quality rendering.
  • Identity Consistency: Both objective spectral measures and subjective listening confirmed that the system successfully maintained a consistent voice identity when switching between English and Chinese.

Conclusion and Future Work

This method effectively "teaches" a monolingual voice to speak a second, phonetically different language. While the results for the English-Chinese pair are promising, the system currently relies on a high-quality reference speaker in the target language. Future research may explore the generalizability of this method to other language pairs and further refine the capture of specific cultural accents and stylistic nuances.

Summary updated

BibTeX
@inproceedings{he2012turning,
  author = {Ji He and Yao Qian and Frank K. Soong and Sheng Zhao},
  title = {{Turning a Monolingual Speaker into Multilingual for a Mixed-language TTS}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=6s1Af86RnnO},
}
Figure