Introduction
In our increasingly globalized digital landscape, "code-switching"—the practice of mixing multiple languages within a single sentence—has become a common feature of communication, appearing in everything from technical documentation to social media. However, conventional Text-to-Speech (TTS) systems struggle to handle mixed-language text naturally.
The standard approach often involves switching between two distinct native voices (e.g., one for Chinese and one for English), which results in a jarring and inconsistent user experience. Alternatively, training an English model using a non-native speaker's recordings often captures unnatural prosody (rhythm and intonation) that sounds "off" to native ears. This paper presents a method to synthesize mixed Chinese and English text using a single, consistent voice that maintains native-level prosody through targeted model adaptation.
Methodology
The researchers utilized an HMM-based Speech Synthesis System (HTS), focusing on Maximum Likelihood Linear Regression (MLLR) for speaker adaptation. The system architecture is designed to bridge the gap between speaker identity and linguistic naturalness.
Key Architectural Innovations
- Cross-Language Adaptation: The system begins with a high-quality model trained on a native English speaker (HOK). Its parameters (spectrum and F0/pitch) are then adapted using a smaller corpus of English data from a target Chinese speaker (ZL), effectively "nudging" the native model to sound like the target speaker.
- The Duration Swap: A critical finding was that adapting duration models (the timing of sounds) to a non-native speaker often degrades quality by capturing their unnatural rhythm. To solve this, the authors retained the native speaker’s original duration models while using the adapted spectrum and pitch models.
- Unified Prosody Tree: To ensure smooth transitions at language boundaries, the system employs a unified prosody structure. This structure ensures that features like "position in sentence" are calculated consistently, preventing the English inserts from sounding like isolated fragments.
Experimental Setup
The study utilized three primary datasets recorded at 16kHz:
- HOK Corpus: 5,000 English sentences from a native English speaker (base model).
- ZL English Corpus: 1,000 English sentences from a Chinese speaker (for speaker adaptation).
- ZL Chinese Corpus: 5,000 Chinese sentences from the same Chinese speaker (base Chinese model).
Results and Discussion
Subjective listening tests were conducted with 9 judges using a Mean Opinion Score (MOS) scale of 1 to 5. The results highlighted the effectiveness of the proposed adaptation strategy.
Key Insights
- Naturalness: The "Adapted System" achieved a score of approximately 3.0, significantly outperforming both the ZL-only baseline (~2.3) and the voice-switching HOK/KOH system (~2.5). This confirms that native-like timing is more important for perceived quality than a perfect identity match in the duration domain.
- Speaker Consistency: The Adapted System maintained high consistency (~3.0), whereas the voice-switching system was rated poorly (~2.0) because listeners clearly perceived two different personas.
- Prosody as the "Soul" of Language: The results suggest that rhythm is harder to correct via adaptation than timbre. By preserving native timing, the system bypasses the prosodic errors typical of non-native speech.
Conclusion
By combining native duration models with speaker-adapted spectral models, this research successfully synthesizes mixed-language text that sounds both natural and consistent. While a small quality gap remains for pure English text due to pitch (F0) contour influences from the adaptation data, the system represents a significant step toward a truly "global voice" for modern TTS applications. Future work will focus on further refining F0 models to better capture native intonation patterns without compromising speaker identity.
Summary updated