Text to speech in new languages without a standardized orthography

Sunayana Sitaram, Gopala Krishna Anumanchipalli, Justin T. Chiu, Alok Parlikar, Alan W. Black

TL;DR

This paper presents a novel method for building Text-to-Speech (TTS) systems for languages lacking a standardized orthography. By leveraging cross-lingual phonetic decoding and iterative acoustic model adaptation, they achieve understandable synthetic voices with surprisingly little data, opening up TTS capabilities for previously underserved languages.

What Problem Does This Paper Address?

Developing speech technologies like TTS for languages typically requires large amounts of transcribed speech data and linguistic resources. However, many of the world's spoken languages lack a standardized writing system, making it impossible to obtain the necessary transcribed data. This creates a significant barrier to deploying useful speech technologies for these languages, which may still have millions of speakers. The authors tackle this challenge by proposing a method to build TTS systems using only audio recordings, without relying on existing text transcriptions.

Methods

The core of their approach is a bootstrapping method that iteratively improves phonetic transcriptions of speech data. Here's a breakdown:

  • Cross-lingual Phonetic Decoding: They start by using an existing acoustic model (from a related language, like English WSJ) and a phoneme language model (from a related language, like German or Marathi) to decode the target language's audio into a sequence of phonemes. This initial decoding is likely noisy.
  • Iterative Acoustic Model Adaptation: The key innovation lies in using these initial phonetic transcripts to train a new, targeted acoustic model for the target language. This process is repeated: the new acoustic model is used to re-decode the audio, generating improved transcripts, which are then used to further refine the acoustic model. This iterative process continues until convergence, aiming to minimize the Mel-Cepstral Distance (MCD), a measure of acoustic similarity.
  • Higher-Level Phonological Unit Induction: Beyond just phonemes, the authors explore inducing more meaningful units to improve prosody and intelligibility. They investigate:
  • Syllabification: Using heuristic rules to group phonemes into syllables.
  • Word Induction: Employing Conditional Random Fields (CRFs) trained on cross-lingual data to identify word-like units from phonetic sequences.
  • Accent Group Discovery: A novel approach that uses acoustic information (pitch contours) to identify "accent groups" – sequences of syllables bearing a single intonational accent. This is a data-driven method that aims to be language-independent.

Datasets

The research utilizes audio data from eight languages:

  • English: ~2 hours (Blizzard Challenge 2013 audiobook task)
  • Dari: ~1 hour (DARPA TRANSTAC project)
  • Iraqi Arabic: ~1 hour (DARPA BOLT project)
  • Pashto: ~39 minutes (DARPA TRANSTAC project)
  • Thai: ~25 minutes (SPICE dataset)
  • Ojibwe: ~12 minutes (Endangered Languages project)
  • Inupiaq: ~5.5 minutes (Endangered Languages project)
  • Konkani: ~5.5 minutes (CMU SPICE project)

The paper notes that the audio data ranged from nearly two hours down to less than six minutes, highlighting their ability to work with limited resources. They also mention using existing corpora for language models (e.g., Europarl for German, Esakal for Marathi).

Key Insights and Ideas

  • Bootstrapping works with minimal data: The iterative acoustic model adaptation is surprisingly effective, yielding understandable synthetic voices with as little as an hour of untranscribed speech. This is a significant breakthrough for low-resource languages.
  • Phonetic transcripts are a viable intermediate: Even without a formal orthography, phonetic strings can serve as a bridge for machine translation or dialog systems, enabling speech output.
  • Higher-level units improve prosody: Simply using phoneme sequences for TTS is suboptimal. Inducing syllables and, more importantly, "accent groups" significantly improves the naturalness and intelligibility of the synthesized speech.
  • Acoustic cues for structure: Discovering higher-level linguistic units like accent groups directly from acoustic features (pitch) is a promising, language-independent approach that bypasses the need for explicit linguistic rules or text.

Key Results

  • Objective Improvement (MCD): The iterative method consistently reduced the Mel-Cepstral Distance (MCD) compared to baseline cross-lingual decoding, especially for languages with more data. For example, English, Dari, and Iraqi Arabic showed significant MCD drops. Even with limited data (e.g., ~12 minutes for Ojibwe), improvements were observed.
  • Subjective Improvement: Listening tests confirmed that voices built using syllabified transcripts were significantly preferred over those using only the best phonetic iteration.
  • Accent Groups Show Promise: Voices synthesized using acoustically derived "accent groups" performed comparably to or better than those using CRF-induced words, and significantly better than baseline phonetic voices. This suggests accent groups are a powerful unit for prosody modeling.
  • Syllables remain reliable: While accent groups showed great promise, syllable-based synthesis remained the most reliable method for improving intelligibility in their current setting, possibly due to data limitations for more complex unit induction.

Limitations and Future Work

  • Data Scarcity for Complex Units: The authors acknowledge that for languages with very little data (like Inupiaq and Konkani), the iterative method struggled to produce significant improvements, and inducing higher-level units like words was less effective.
  • Language Model Adaptation: While they kept the language model consistent during iterations, they note that adapting it could be a future extension.
  • Phoneset Pruning: They plan to explore more principled methods for pruning the initial phoneset.
  • Machine Translation Integration: A key future direction is to extend this work to use machine translation to provide a usable writing system for languages without standardized orthography.
  • Exploring Other Acoustic/Language Models: They aim to investigate combinations of multiple acoustic and language models for the initial decoding pass.

Summary updated

BibTeX
@inproceedings{sitaram2013text,
  author = {Sunayana Sitaram and Gopala Krishna Anumanchipalli and Justin T. Chiu and Alok Parlikar and Alan W. Black},
  title = {{Text to speech in new languages without a standardized orthography}},
  booktitle = {SSW},
  year = {2013},
}
Figure