Unsupervised Phonetic and Word Level Discovery for Speech to Speech Translation for Unwritten Languages

Steven Hillis, Anushree Prasanna Kumar, Alan W. BlackInterspeech 2019 · DOI

Introduction

Modern speech-to-speech (S2S) translation systems typically rely on a pipeline of Automatic Speech Recognition (ASR), Machine Translation (MT), and Text-to-Speech (TTS) synthesis. However, this paradigm fails for the thousands of unwritten languages that lack standardized scripts and textual training data. Researchers at Carnegie Mellon University have proposed a novel solution: automatically discovering a symbolic representation directly from raw audio to serve as a proxy for text. By clustering phonetic units into higher-order, word-like structures, they demonstrate that translation is possible even without manual transcription.

Methodology: From Audio to Discovered Words

The proposed approach involves a two-stage unsupervised pipeline:

1. Phonetic Discovery

Initially, acoustic models are built iteratively using only audio data. A "seed" model (trained on English Wall Street Journal data) generates rough initial transcripts. These transcripts are then used to build a synthetic voice. The phonetic transcripts are refined over ten iterations, with the optimal version selected based on the lowest Mel Cepstral Distortion (MCD), indicating the most "natural" synthetic speech.

2. Word Discovery through Clustering

Raw phonetic units often lack sufficient semantic density for effective translation. To create more "word-like" units, the authors evaluated three unsupervised clustering techniques:

  • Byte Pair Encoding (BPE): Iteratively merges the most frequent adjacent symbols.
  • N-gram Clustering: Merges the top $k$ most frequent sequences of phones over $p$ passes.
  • Goldwater Approach: A Bayesian model using a Dirichlet process for word segmentation.
Examples of different symbolic representations discovered from audio compared to text-based units.

The resulting discovered units are then used to train a Moses Statistical Machine Translation model to translate into English.

Experimental Setup

The study utilized the CMU Wilderness dataset, focusing on five low-resource African languages: Avatime (AVN), Oromo (ORM), Hausa (HAU), Masaba (MYX), and Lunyole (NUJ). A parallel corpus of English Bible data served as the translation target.

Results and Discussion

The core finding of the research is that "semantic density" is critical for translation quality. Translating raw phone sequences is highly ineffective; however, clustering these units into "discovered words" provides the MT model with the necessary context to identify patterns.

BLEU-1 scores across different discovery methods and languages, showing the impact of clustering.

As shown in the results, raw audio-based phones achieved a mean BLEU-1 score of only 0.05. N-gram clustering emerged as the most effective method, increasing the mean score to 0.13. Notably, in languages like Masaba (MYX) and Hausa (HAU), these unsupervised N-gram units performed comparably to, or even better than, phones derived from actual text.

Limitations and Future Outlook

While promising, the approach faces challenges such as data mismatch between audio sources and target text, and the small size of the parallel corpora (approximately 3,000 verses per language). Furthermore, language-specific phonetic structures can lead to anomalies, as seen with Avatime (AVN), where clustering did not improve performance.

Future research aims to create a "discovered lexicon" to train language models, which could then be used as a prior to further refine the phone discovery process in a self-improving cycle. This work paves the way for bringing modern translation technology to the world's unwritten and under-resourced languages.

Summary updated

BibTeX
@inproceedings{hillis2019unsupervised,
  author = {Steven Hillis and Anushree Prasanna Kumar and Alan W. Black},
  title = {{Unsupervised Phonetic and Word Level Discovery for Speech to Speech Translation for Unwritten Languages}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-3026},
}
Figure