Multilingual Speech Recognition for Information Retrieval in Indian Context

N. Udhaykumar, S.k. Ramakrishnan, R. SwaminathanNAACL 2004

TL;DR

This paper tackles the challenge of building robust multilingual speech recognition systems for Indian languages, specifically focusing on Hindi, Tamil, and accented English. They introduce novel techniques like cross-lingual bootstrapping and automatic grapheme-to-phoneme conversion to overcome data scarcity and linguistic complexities, paving the way for more inclusive spoken language information retrieval.

What Problem Does This Paper Address?

The increasing prevalence of spoken language technologies for information retrieval necessitates systems that can understand diverse users. For a country like India, with its rich linguistic tapestry and widespread use of multiple languages within a single conversation, this presents a significant hurdle. Traditional speech recognition systems are often language-specific and struggle with:

  • Language Switching: Indian speakers frequently mix languages within a single utterance.
  • Accented Speech: Non-native speakers, particularly those using English as a second language, exhibit accents that degrade recognition accuracy.
  • Data Scarcity: Developing high-performance speech recognition models for low-resource languages requires substantial amounts of transcribed speech data, which is often unavailable.
  • Out-of-Vocabulary (OOV) Words: Handling words not present in the system's lexicon is crucial for real-world applications.

This paper aims to address these issues by developing a unified multilingual speech recognition framework that can handle code-switching, adapt to accented speech, and efficiently manage linguistic variations.

Methods

The authors employ a multi-pronged approach to build their multilingual speech recognition system:

  • Monolingual Baseline Systems: They start by building independent speech recognition systems for Hindi and Tamil using the HTK toolkit and the OGI Multilanguage Telephone Speech Corpus.
  • Context-Dependent Training: To improve accuracy, they move from context-independent monophones to context-dependent triphones, utilizing decision trees for phonetic clustering and Maximum Likelihood Linear Regression (MLLR) for adaptation.
  • Multilingual Phoneme Set: A key innovation is the creation of a shared multilingual phoneme set by combining acoustically similar phones from different languages. This leverages model sharing for efficiency and data sharing for improved parameter estimation, especially for less frequent phonemes. They use IPA mapping to create this unified phoneme pool.
  • Cross-Lingual Bootstrapping: To combat data scarcity for target languages (like accented English), they initialize recognizers using pre-trained models from other languages (e.g., English models seeded with Tamil data). This bootstrapped system is then retrained with the target language's data.
  • Automatic Grapheme-to-Phoneme (G2P) Conversion: To handle OOV words and reduce lexicon size, they explore both rule-based and data-driven (CART - Classification And Regression Trees) approaches for converting written text to phonetic transcriptions. This is crucial for generating pronunciations for words not explicitly in the dictionary.
  • Accented English Adaptation: They investigate techniques like MLLR adaptation and retraining with native language data to improve recognition of accented English.

Datasets

  • OGI Multilanguage Telephone Speech Corpus: Used for training and testing the monolingual Hindi and Tamil systems. This corpus contains spontaneous speech from various callers.
  • Network Speech Database (NTIMIT): A telephone-bandwidth version of the TIMIT database, used for training English models.
  • Foreign Accented English Corpus (CSLU): Used for experiments with accented English.
  • Native American Accented Network TIMIT Corpus: Also used for experiments with accented English.
  • Tamil Handcrafted Lexicon: A 2k word lexicon used for evaluating the G2P systems.

Key Insights and Ideas

  • IPA Mapping for Multilingualism: Using the International Phonetic Association (IPA) as a bridge to combine phonemes across languages is a powerful strategy for building compact and efficient multilingual models. It leverages linguistic universals.
  • Bootstrapping for Low-Resource Languages: Cross-lingual bootstrapping is a highly effective technique for kick-starting speech recognition development in languages with limited data, significantly improving initial performance compared to training from scratch.
  • CART for G2P: Data-driven approaches like CART can automatically learn complex grapheme-to-phoneme conversion rules, potentially outperforming manually crafted rules and discovering intricate linguistic patterns.
  • Native Language Influence on Accents: The perception of target phones in accented speech is closely tied to the speaker's native language phonology. This suggests that adapting native language models is a more promising avenue than direct adaptation from a different language.

Key Results

  • Monolingual Accuracy: Baseline monolingual systems achieved word-level accuracies of around 49.7% for Hindi and 50.3% for Tamil. After lexicon improvements, Tamil accuracy rose to 56% and Hindi to 52.9%.
  • Multilingual System Performance: The multilingual system achieved a respectable 90.3% accuracy.
  • Bootstrapped System Improvement: Bootstrapping significantly boosted the multilingual system's accuracy to 94.5%.
  • G2P System Performance: Both rule-based and CART-based G2P systems achieved high word accuracies (95.2% and 96.3% respectively) on a Tamil lexicon, with CART showing a slight edge.
  • Accented English Adaptation: MLLR adaptation and retraining with native language data showed improved accuracy for accented English recognition compared to a baseline system trained on native English.

Limitations and Future Work

  • Spontaneous Speech Effects: The paper acknowledges that handling spontaneous speech effects (filled pauses, laughter, etc.) and nonverbal sounds remains a significant challenge.
  • Model Interpolation and FSTs: Future work includes exploring model interpolation and weighted finite state transducers (FSTs) to further improve performance, especially on accented English.
  • Data Collection: The authors emphasize the need for more data ("there is no data like more data, there is also no data like real data") and have initiated data collection for future experiments.
  • Language Identification: While LID is mentioned as a component, its detailed integration and performance within the full system could be further explored.

Summary updated

BibTeX
@inproceedings{udhaykumar2004multilingual,
  author = {N. Udhaykumar and S.K. Ramakrishnan and R. Swaminathan},
  title = {{Multilingual Speech Recognition for Information Retrieval in Indian Context}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure