Advances in the CMU/Interact Arabic GALE Transcription System

Mohamed Noamany, Thomas Schaaf, Tanja SchultzNAACL 2007

TL;DR

This paper details the CMU/InterACT system's significant improvements in Arabic Automatic Speech Recognition (ASR) for broadcast news and conversations. By combining vowelized and non-vowelized systems, leveraging web-scraped TV transcripts for training, and adapting acoustic and language models, they achieved a 40% relative improvement over their previous system.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) for Arabic presents unique challenges, particularly with Modern Standard Arabic (MSA). One major hurdle is the omission of short vowels in written text, which are crucial for conveying grammatical case and meaning. This ambiguity makes it difficult for ASR systems to accurately transcribe spoken Arabic. Furthermore, obtaining sufficient and diverse training data, especially for conversational speech, is a persistent challenge in ASR research. The GALE (Global Autonomous Language Exploitation) program aims to address these issues by developing technologies to process large volumes of multilingual speech and text, with NIST conducting regular evaluations. This paper focuses on the CMU/InterACT team's efforts to enhance their Arabic ASR system for the GALE 2006 evaluation, specifically tackling the vowelization problem and data scarcity.

Methods

The CMU/InterACT system is built upon the Janus Recognition Toolkit (JRTk) and the IBIS decoder. Key methodological advancements include:

  • Vowelized System: To address the missing vowel issue, they developed a vowelized pronunciation dictionary. This involved using a Buckwalter morphological analyzer and supplementing it with data from the LDC Arabic Tree-Bank. When words couldn't be vowelized by these tools, they defaulted to the written non-vowelized form. This resulted in an average of 5 pronunciations per word, which were then pruned using forced alignment to reduce variants.
  • Non-Vowelized System: A grapheme-to-phoneme approach was used to automatically generate pronunciations. This system also incorporated word-boundary tags into the pronunciation dictionary to aid decision tree clustering.
  • Combined System: A significant improvement was achieved by combining the vowelized and non-vowelized systems. This was implemented using a 3-pass decoding strategy: the first pass used the speaker-independent (SI) vowelized system, the second pass used the speaker-adaptive (SA) non-vowelized system, and the third pass used the SA vowelized system.
  • Slightly Supervised Training with Web Data: To augment conversational training data, they collected transcripts from Al-Jazeera TV shows available online. These transcripts, though lacking timestamps, were used for "slightly supervised" training. This involved recording TV shows, extracting transcripts, processing them (converting HTML to text, numbers to words, removing non-Arabic words), and then aligning them with the decoder output to select usable segments.
  • Acoustic Model Adaptation: The system employs a multi-pass decoding strategy that includes speaker-independent acoustic models, followed by VTLN (vocal tract length normalization) and feature space adaptation (FSA) using CMLLR (constrained MLLR) transforms. Model space adaptation is then performed using maximum likelihood linear regression.
  • Language Model Adaptation: Multiple 4-gram language models were built from various data sources (broadcast news, Al-Jazeera web data) and then interpolated. The interpolation weights were determined based on held-out data, with Al-Jazeera data receiving the highest weight.

Datasets

  • Training Data:
    • 40 hours of Broadcast News (BN) from manually transcribed FBIS data.
  • 50 hours of BN from LDC-TDT4 (selected from 85 hours using a slightly supervised approach).
  • 30 hours of Broadcast Conversation (BC) recorded from Al-jazeera TV.
  • 70 hours of LDC-GALE data (40 hours BN, 30 hours BC).
  • 750 hours of BN created and shared by IBM (unsupervised training).
  • 50 hours of Broadcast Conversation (BC) from Al-jazeera TV (web transcripts).
  • Evaluation Data: RT04 (GALE evaluation set).

Key Insights and Ideas

  • Vowelization is Key: Explicitly modeling vowels, even with the complexity of multiple pronunciations per word, significantly improves Arabic ASR performance. This highlights the importance of addressing inherent linguistic ambiguities in the input.
  • Web Data is a Goldmine for Conversations: Leveraging publicly available web transcripts from TV stations is an effective and cost-efficient way to obtain much-needed conversational speech data, which is often scarce.
  • Combining Systems Yields Synergy: Merging a vowelized system with a non-vowelized system, through a multi-pass decoding strategy, leads to greater gains than either system alone. This suggests that different approaches capture complementary information.
  • Slight Supervision is Powerful: The "slightly supervised" approach, where transcripts are aligned with recorded audio, allows for the utilization of imperfect but abundant web data, leading to substantial improvements.

Key Results

  • Overall Improvement: The CMU/InterACT system achieved a 40% relative improvement compared to their legacy system.
  • Vowelized System Gains: The vowelized system achieved a 1.3% absolute gain on the SI pass and a 1.5% absolute gain on the SA pass over the non-vowelized system on the RT04 set.
  • Combined System Gains: Combining the vowelized and non-vowelized systems resulted in a 4.0% relative improvement over the vowelized system alone.
  • Web Transcript Impact: Using 30 hours of web-transcribed data resulted in a 7% relative improvement on RT04.
  • Real-time Behavior: The system's real-time behavior improved from 20RT to 10RT with a minimal loss of 0.2% in performance.

Limitations and Future Work

  • Unresolved Vowelizations: Approximately 11,000 entries in the vocabulary could not be vowelized by the morphological analyzer or the Tree-Bank. The authors plan to explore adding weights to different multiple pronunciations and adding vowelization for these remaining words.
  • Estimating Probability Priors: The authors envision even higher improvements by estimating and applying probability priors to multiple pronunciations.
  • Data Quality: While web transcripts were valuable, they lacked timestamps and sometimes omitted commercials or news breaks. The process of aligning and selecting usable segments was crucial.
  • Further Data Exploration: The authors are continuing to explore the use of web data and other methods to address data scarcity.

Summary updated

BibTeX
@inproceedings{noamany2007advances,
  author = {Mohamed Noamany and Thomas Schaaf and Tanja Schultz},
  title = {{Advances in the CMU/Interact Arabic GALE Transcription System}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure