Rapid bootstrapping of five eastern european languages using the rapid language adaptation toolkit

Ngoc Thang Vu, Tim Schlippe, Franziska Kraus, Tanja SchultzInterspeech 2010

TL;DR

This paper presents a toolkit (RLAT) for rapidly building speech recognition systems for low-resource languages. By leveraging web crawling and clever language model adaptation techniques, they demonstrate the ability to create functional systems for five Eastern European languages in just twenty days with minimal human effort. This work significantly lowers the barrier to entry for developing speech technologies for a wider range of languages.

What Problem Does This Paper Address?

Developing automatic speech recognition (ASR) systems for new languages is a challenging and time-consuming process. While ASR technology has advanced significantly, most research and development efforts are concentrated on a few high-resource languages with large speaker populations and significant IT needs. The vast majority of the world's languages, however, remain underserved. The core problem this paper addresses is how to efficiently and cost-effectively port ASR technology to these low-resource languages, specifically focusing on five Eastern European Slavic languages (Bulgarian, Croatian, Czech, Polish, and Russian). The authors aim to minimize human effort and time while maximizing the use of readily available, albeit noisy, text data from the internet.

Methods

The paper introduces and utilizes the Rapid Language Adaptation Toolkit (RLAT), a system designed to streamline the development of speech processing models. Key methods include:

  • Web Crawling for Text Data: RLAT employs a "snapshot" function to crawl large amounts of text data from the internet. This process involves following links up to a specified depth (e.g., 20) to gather diverse content.
  • Rapid Bootstrapping: For initial acoustic models, RLAT uses a multilingual acoustic model inventory trained on seven other GlobalPhone languages. It selects the closest matching acoustic models as seeds for the new language based on an IPA-based phone mapping.
  • "Quick and Dirty" Text Processing: Collected text data is initially processed by removing HTML tags, code fragments, and empty lines. This raw text is then used to build language models.
  • Vocabulary Selection Strategy: To manage the large vocabulary growth inherent in morphologically rich Slavic languages, a strategy is employed to select the 100K most frequent words. The threshold for word inclusion is dynamically adjusted based on word frequency and vocabulary growth.
  • Text Normalization: To mitigate noise from web crawling, several text normalization steps are applied: deleting special characters, converting digits/numbers/dates to text, removing punctuation, and converting all text to lowercase.
  • Language Model Interpolation: To smooth perplexity variations and improve performance, a linear interpolation scheme is used. Daily language models built from crawled data are combined to form a final, more robust language model.
  • Data Diversity Enhancement: The impact of using multiple websites as starting points for crawling is investigated to improve robustness and data quality.

Datasets

  • GlobalPhone Speech Corpus: This multilingual corpus, covering 20 languages, was used for speech data. For the five target Eastern European languages, it provided speech data from approximately 100 native speakers per language, with transcriptions of read texts from news and economic articles.
  • Web-crawled Text Data: For each of the five languages, text data was collected from specific websites (e.g., dariknews.bg for Bulgarian, www.lidovky.cz for Czech). The size of these corpora varied, ranging from tens to hundreds of millions of words.

Key Insights and Ideas

  • Web crawling is a viable, low-cost source for low-resource language data: The paper demonstrates that vast amounts of text data can be efficiently gathered from the internet, significantly reducing the cost and effort compared to traditional corpus creation.
  • Rapid bootstrapping with multilingual models is effective: Leveraging pre-trained acoustic models from related languages allows for the quick creation of initial ASR systems, even for languages with limited existing resources.
  • Text normalization is crucial for web-crawled data: The noisy nature of web text necessitates robust normalization techniques to improve language model quality and reduce word error rates.
  • Language model interpolation smooths out noise and improves performance: Combining daily language models built from incrementally crawled data helps to manage perplexity increases and leads to better recognition results.
  • Slavic languages present unique challenges: The rich morphology of these languages leads to large vocabulary growth and high out-of-vocabulary (OOV) rates, requiring specific strategies for vocabulary management.

Key Results

  • Rapid System Development: Initial ASR systems for five Eastern European languages were built within twenty days using RLAT, with minimal human effort.
  • Word Error Rates (WER): On the GlobalPhone evaluation set, the final systems achieved the following WERs:
  • Bulgarian: 16.9%
    • Croatian: 32.8%
    • Czech: 23.5%
    • Polish: 20.4%
    • Russian: 36.2%
  • Text Normalization Impact: Post-processing steps led to significant relative WER reductions, ranging from 6% (Russian) to 15% (Bulgarian).
  • Data Diversity Benefits: Incorporating text from additional websites improved language model quality, notably reducing perplexity and OOV rates, especially for Croatian.

Limitations and Future Work

  • Text Quality and Noise: Despite normalization, the inherent noise and varying quality of web-crawled text remain a challenge, leading to perplexity increases over time.
  • Vocabulary Management: While a strategy was employed, managing the large vocabulary of morphologically rich languages is an ongoing challenge.
  • Suboptimal Crawling: The reliance on a single starting website can make the crawling process fragile, as demonstrated by the premature termination of the Croatian crawl.
  • Future Work: The authors suggest further investigation into text data diversity and the impact of different crawling strategies. Optimizing vocabulary selection and exploring more advanced language model adaptation techniques are also implied directions.

Summary updated

BibTeX
@inproceedings{vu2010rapid,
  author = {Ngoc Thang Vu and Tim Schlippe and Franziska Kraus and Tanja Schultz},
  title = {{Rapid bootstrapping of five eastern european languages using the rapid language adaptation toolkit}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure