Integration of language identification into a recognition system for spoken conversations containing code-Switches

Jochen Weiner, Ngoc Thang Vu, Dominic Telaar, Florian Metze, Tanja Schultz, Dau-Cheng Lyu, Engsiong Chng, Haizhou Li

TL;DR

This paper tackles the challenge of accurately transcribing spoken conversations that mix languages (code-switching), specifically Mandarin and English. They propose two novel methods to integrate language identification (LID) directly into the speech recognition system, showing significant improvements in accuracy when the LID component performs well.

What Problem Does This Paper Address?

Code-switching, the practice of alternating between two or more languages within a single conversation, is incredibly common in multilingual communities. However, building automatic speech recognition (ASR) systems that can handle this phenomenon is exceptionally difficult. This is because code-switching complicates acoustic modeling and language modeling, and existing approaches often rely on separate, potentially error-prone language identification (LID) modules. The authors aim to bridge this gap by directly integrating LID information into the ASR decoding process to improve recognition accuracy for code-switched speech.

Methods

The paper introduces two main approaches for integrating LID into a multilingual ASR system:

  • Multistream Approach: This is a straightforward method where the LID score (representing the probability of a given language at a specific time frame) is combined with the acoustic model score. This combination happens at the frame level, effectively creating an additional "stream" of information for the ASR decoder. A weighting factor determines how much influence the LID score has.
  • Language Lookahead Approach: This is an extension of the multistream approach. It not only considers the LID score for the current frame but also incorporates LID scores from subsequent frames. The idea is to use future language information to help correct potential misclassifications of the current frame, making the recognition more robust.

Both methods are evaluated by controlling the accuracy of the LID component itself, allowing the researchers to study the impact of LID performance on the overall ASR system.

Datasets

The primary dataset used in this research is the SEAME corpus. This corpus consists of approximately 63 hours of recorded Mandarin-English code-switched speech from Singaporean and Malaysian speakers. It's specifically designed for research on spontaneous code-switching and includes word-level transcriptions with language boundary alignments. The corpus is divided into training, development, and test sets.

Key Insights and Ideas

  • Direct Integration is Key: Simply relying on separate LID and ASR systems can be problematic because errors in LID are difficult to recover. Integrating LID directly into the ASR decoder offers a more robust solution.
  • "Language Lookahead" Offers an Edge: While both multistream and lookahead approaches improve performance, the lookahead method, by considering future language context, shows promise for further enhancing robustness.
  • LID Accuracy is Crucial: The effectiveness of both proposed methods is directly tied to the accuracy of the LID component. As LID accuracy increases, ASR performance improves almost linearly.
  • A Sweet Spot for LID: The research suggests that achieving a minimum LID frame accuracy of 85% is a critical threshold for realizing significant ASR performance gains.

Key Results

The paper demonstrates that both the Multistream and Language Lookahead approaches lead to improvements in ASR performance for code-switched speech compared to a baseline system.

  • When the LID component achieves a minimum frame accuracy of 85%, both approaches yield at least a 4% relative improvement in Mixed Error Rate (MER).
  • For an "oracle" scenario (where LID is assumed to be 100% accurate), the Language Lookahead approach achieved an MER of 34.2%, and the Multistream approach achieved 34.4%, both significantly better than the baseline MER of 36.9%.
  • The paper also notes that the frame error rate for voice activity detection was 5.88%, and the initial LID frame accuracy on the development set was 70.64%.

Limitations and Future Work

  • Controlled LID: The experiments relied on artificially controlled LID accuracies rather than solely on the actual LID system's output. While this allowed for a controlled study, future work could focus on improving the LID system itself to reach higher accuracies.
  • Weight Optimization: The weights for combining LID and acoustic scores were experimentally optimized. Further research could explore more adaptive or dynamic weighting schemes.
  • Generalizability: The study focused on Mandarin-English code-switching. Extending these approaches to other language pairs and more complex code-switching scenarios would be valuable.
  • Impact of LID Errors: While the paper shows the impact of LID accuracy, a deeper analysis of how specific types of LID errors affect ASR performance could be beneficial.

Summary updated

Abstract

This paper describes the integration of language identification (LID) into a multilingual automatic speech recognition (ASR) system for spoken conversations containing code-switches between Mandarin and English. We apply a multistream approach to combine at frame level the acoustic model score and the language information, where the latter is provided by an LID component. Furthermore, we advance this multistream approach by a new method called “Language Lookahead”, in which the language information of subsequent frames is used to improve accuracy. Both methods are evaluated using a set of controlled LID results with varying frame accuracies. Our results show that both approaches improve the ASR performance by at least 4% relative if the LID achieves a minimum frame accuracy of 85%.
BibTeX
@inproceedings{weiner2012integration,
  author = {Jochen Weiner and Ngoc Thang Vu and Dominic Telaar and Florian Metze and Tanja Schultz and Dau-Cheng Lyu and Engsiong Chng and Haizhou Li},
  title = {{Integration of language identification into a recognition system for spoken conversations containing code-Switches}},
  booktitle = {SLTU},
  year = {2012},
}

Loading related content...

Figure