HMM based TTS for mixed language text

Zhiwei Shuang, Shiyin Kang, Yong Qin, Li-Rong Dai, Lianhong CaiInterspeech 2010 · DOI

Introduction

In our increasingly globalized digital landscape, "code-switching"—the practice of mixing multiple languages within a single sentence—has become a common feature of communication, appearing in everything from technical documentation to social media. However, conventional Text-to-Speech (TTS) systems struggle to handle mixed-language text naturally.

The standard approach often involves switching between two distinct native voices (e.g., one for Chinese and one for English), which results in a jarring and inconsistent user experience. Alternatively, training an English model using a non-native speaker's recordings often captures unnatural prosody (rhythm and intonation) that sounds "off" to native ears. This paper presents a method to synthesize mixed Chinese and English text using a single, consistent voice that maintains native-level prosody through targeted model adaptation.

Methodology

The researchers utilized an HMM-based Speech Synthesis System (HTS), focusing on Maximum Likelihood Linear Regression (MLLR) for speaker adaptation. The system architecture is designed to bridge the gap between speaker identity and linguistic naturalness.

The overall system architecture, illustrating the parallel training and synthesis paths for English and Chinese components.

Key Architectural Innovations

  1. Cross-Language Adaptation: The system begins with a high-quality model trained on a native English speaker (HOK). Its parameters (spectrum and F0/pitch) are then adapted using a smaller corpus of English data from a target Chinese speaker (ZL), effectively "nudging" the native model to sound like the target speaker.
  2. The Duration Swap: A critical finding was that adapting duration models (the timing of sounds) to a non-native speaker often degrades quality by capturing their unnatural rhythm. To solve this, the authors retained the native speaker’s original duration models while using the adapted spectrum and pitch models.
  3. Unified Prosody Tree: To ensure smooth transitions at language boundaries, the system employs a unified prosody structure. This structure ensures that features like "position in sentence" are calculated consistently, preventing the English inserts from sounding like isolated fragments.

Experimental Setup

The study utilized three primary datasets recorded at 16kHz:

  • HOK Corpus: 5,000 English sentences from a native English speaker (base model).
  • ZL English Corpus: 1,000 English sentences from a Chinese speaker (for speaker adaptation).
  • ZL Chinese Corpus: 5,000 Chinese sentences from the same Chinese speaker (base Chinese model).

Results and Discussion

Subjective listening tests were conducted with 9 judges using a Mean Opinion Score (MOS) scale of 1 to 5. The results highlighted the effectiveness of the proposed adaptation strategy.

Mean Opinion Scores (MOS) for naturalness in mixed-language text, comparing the baseline ZL system, the voice-switching KOH system, and the proposed Adapted system.

Key Insights

  • Naturalness: The "Adapted System" achieved a score of approximately 3.0, significantly outperforming both the ZL-only baseline (~2.3) and the voice-switching HOK/KOH system (~2.5). This confirms that native-like timing is more important for perceived quality than a perfect identity match in the duration domain.
  • Speaker Consistency: The Adapted System maintained high consistency (~3.0), whereas the voice-switching system was rated poorly (~2.0) because listeners clearly perceived two different personas.
  • Prosody as the "Soul" of Language: The results suggest that rhythm is harder to correct via adaptation than timbre. By preserving native timing, the system bypasses the prosodic errors typical of non-native speech.

Conclusion

By combining native duration models with speaker-adapted spectral models, this research successfully synthesizes mixed-language text that sounds both natural and consistent. While a small quality gap remains for pure English text due to pitch (F0) contour influences from the adaptation data, the system represents a significant step toward a truly "global voice" for modern TTS applications. Future work will focus on further refining F0 models to better capture native intonation patterns without compromising speaker identity.

Summary updated

BibTeX
@inproceedings{shuang2010hmm,
  author = {Zhiwei Shuang and Shiyin Kang and Yong Qin and Li-Rong Dai and Lianhong Cai},
  title = {{HMM based TTS for mixed language text}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-237},
}
Figure