A general approach to TTS reading of mixed-language texts

Leonardo Badino, Claudia Barolo, Silvia QuazzaInterspeech 2004

TL;DR

This paper introduces a novel "Foreign Pronunciation Strategy" for Text-to-Speech (TTS) systems that allows a single-language voice to convincingly read text containing foreign words or phrases. By employing a language-independent phoneme mapping algorithm based on articulatory features, the system achieves plausible, albeit accented, foreign pronunciation without requiring multilingual speakers or separate voice databases.

What Problem Does This Paper Address?

Modern communication often involves texts that seamlessly blend multiple languages, such as emails, online content, or even everyday conversations. Traditional Text-to-Speech (TTS) systems are typically designed for monolingual texts, relying on single-language acoustic databases and language-specific analysis. This poses a significant challenge when encountering mixed-language input, as switching to a different voice for each language can be impractical and disruptive. The core problem is enabling a single TTS voice to handle these mixed-language scenarios gracefully, producing understandable and natural-sounding output.

Methods

The paper proposes a "Foreign Pronunciation Strategy" that leverages the existing flexible, modular architecture of the Loquendo TTS system. The strategy involves two main approaches:

  1. Bilingual Acoustic Dictionaries: For specific language pairs (e.g., Castilian-Catalan), bilingual dictionaries are created using speakers fluent in both languages. This allows for more accurate pronunciation of foreign segments.
  2. Foreign Pronunciation Strategy (FPP): This is the paper's primary contribution. It allows a voice native to language L1 to pronounce text in language L2. The process involves:
  3. Language Guessing: A module automatically detects the language of text segments.
  4. Foreign Text Analysis: The L2 portion of the text is analyzed to obtain its broad phonetic transcription.
  5. Phoneme Mapping Algorithm: This is the core innovation. It takes the L2 phoneme string and maps it to the closest L1 phonemes using a language-independent algorithm. This algorithm relies on a Phonetic Similarity Function that represents phonemes as vectors of articulatory features (e.g., manner, place of articulation). A similarity score is computed based on the distance between these feature vectors. Features are weighted based on their perceptual importance, and an iterative process involving perceptual tests was used to refine these weights and distances.
  6. L1 Prosody and Synthesis: Once mapped, the foreign phonemes are integrated into the L1 phonetic transcription, and L1 prosodic rules and acoustic units are applied for synthesis. This ensures coherence with the L1 acoustic database.

Datasets

The paper mentions that the Foreign Pronunciation Strategy was tested and refined using languages including Catalan, Chinese, English, French, German, Greek, Italian, Spanish, and Swedish. However, specific details about the size or nature of the acoustic databases or word lists used for training the Language Guesser are not provided. No new datasets were introduced.

Key Insights and Ideas

  • Approximate Pronunciation is Often Desirable: The authors argue that a perfectly accurate foreign pronunciation might not always be natural or preferred. Speakers often adapt foreign words to their native phonological system, resulting in an "accent." This "approximate pronunciation" can be more realistic and even desirable in certain contexts.
  • Language-Independent Phoneme Mapping: The core idea of using articulatory features to define phonetic similarity allows for a general-purpose algorithm that doesn't require language-specific tables, making it scalable to new languages.
  • Feature-Based Similarity for Perception: The hypothesis that phonemes with similar articulatory features are perceived as similar is a strong, yet effective, basis for building a robust mapping algorithm.
  • Strategic Switching Point: The paper suggests that the switch to the native language of the voice should occur just before the fine co-articulation and allophone modification stage, preserving the native speaker's "performance" while incorporating foreign phonetic elements.

Key Results

The paper demonstrates the effectiveness of the Foreign Pronunciation Strategy through an example (Figure 2) showing a French phrase embedded in an English sentence. The output is described as "plausible" and capturing "the typical behavior of foreign speakers, resulting almost in a caricature of the various foreign accents." The authors state that the Phoneme Similarity function obtains "satisfying results, in most cases yielding the same phoneme mapping that would have been manually selected." The approach is noted to be efficient and language-independent, facilitating the development of multilingual TTS systems.

Limitations and Future Work

  • Limitations:
    • The approach provides "approximate pronunciations" which, while often desirable, might not meet the highest quality standards for all foreign language segments.
  • Tonal languages are not well-handled, as tones are not represented in the current feature vectors.
  • The quality of foreign pronunciation can be affected by the availability of appropriate acoustic units in the L1 database, potentially leading to discontinuities if rare phoneme sequences arise.
  • The current implementation might not perfectly capture all nuances of phonetic perception, which can be influenced by cultural factors.
  • Future Work:
    • Augmenting voice databases with speech material specifically designed for foreign pronunciation, especially for challenging phoneme sequences or languages.
  • Enriching voices to provide smoother foreign pronunciation, particularly for English.
  • Potentially incorporating cultural aspects related to grapheme representation and language evolution into the similarity definition.

Summary updated

BibTeX
@inproceedings{badino2004general,
  author = {Leonardo Badino and Claudia Barolo and Silvia Quazza},
  title = {{A general approach to TTS reading of mixed-language texts}},
  booktitle = {Interspeech},
  year = {2004},
}
Figure