TL;DR
This paper tackles the challenge of improving Automatic Speech Recognition (ASR) for code-switching (CS) speech, particularly for under-resourced languages like Frisian. The authors demonstrate that augmenting both acoustic and language models with data from the high-resource language (Dutch) and synthetically generated CS text significantly boosts ASR performance.
What Problem Does This Paper Address?
Code-switching (CS), the practice of alternating between two or more languages within a single conversation, is common in multilingual communities but poses a significant challenge for ASR systems. Traditional ASR systems are typically trained on monolingual data, and adapting them to CS speech is difficult due to the scarcity of labeled CS training data. This is especially true when one of the languages involved is under-resourced, like Frisian, which has limited available speech and text data. Existing approaches often struggle with language identification errors or require extensive manual annotation. This paper aims to improve CS ASR by effectively leveraging available data, including monolingual data from the high-resource language and automatically generated CS text.
Methods
The authors employ a two-pronged approach to enhance both the acoustic model (AM) and the language model (LM) for CS ASR:
Acoustic Modeling (AM): * Leveraging Monolingual Data: They investigate how to incorporate monolingual Dutch speech data into the training of state-of-the-art AMs. Previous attempts were hindered by data imbalance, leading to performance degradation. This work builds upon prior efforts to increase the amount of automatically transcribed CS speech data, making the inclusion of Dutch data more viable. * Accent Robustness: To address the issue of Dutch spoken with a Frisian accent, they include Flemish Dutch speech data, aiming to improve the accent-robustness of the acoustic models. * Data Augmentation: They utilize a 3-fold data augmentation technique to further enhance the AM training.
Language Modeling (LM): * Synthetic CS Text Generation: Since naturally occurring CS text is scarce, they generate synthetic CS text through several methods: * Recurrent Language Models (LMs): Training LSTM-based LMs on existing, albeit limited, CS transcriptions and then using these models to generate larger amounts of CS text. * Machine Translation: Translating Dutch text extracted from large Dutch speech corpora. This method is expected to introduce CS examples, particularly with proper nouns. * Automatic Transcriptions: Incorporating transcriptions generated by previous automatic CS speech recognition strategies. * LM Rescoring: They use RNN-LMs trained on the enriched CS text for lattice rescoring, further refining the language model's performance.
Datasets
- FAME! Corpus: This is the primary dataset for this research, containing Frisian-Dutch radio broadcasts. It includes manually annotated data (8.5 hours of Frisian, 3 hours of Dutch) and automatically annotated data ('Frisian Broadcast' data, 125.5 hours). Development and test sets are also derived from this corpus.
- Spoken Dutch Corpus (CGN): This large corpus of monolingual Dutch speech (442.5 hours) and Flemish Dutch speech (307.5 hours) is used to augment the acoustic models.
- Baseline Text Corpus: A bilingual text corpus of 37M Frisian and 8.8M Dutch words, primarily from monolingual sources.
Key Insights and Ideas
- Data Imbalance Mitigation is Key: The success of incorporating high-resource monolingual data into low-resource CS ASR hinges on first significantly increasing the amount of in-domain CS training data. This makes the inclusion of Dutch data less detrimental.
- Synthetic Data is Powerful: Generating synthetic CS text, especially through LMs trained on existing CS data and machine translation, is a viable strategy to overcome the scarcity of natural CS text for LM training.
- Accent Matters: Including Flemish Dutch data helped improve the robustness of the acoustic models to accents present in the target CS speech.
- Combined Augmentation Yields Best Results: The most significant improvements in both perplexity and ASR performance were achieved by combining acoustic and language model augmentation strategies.
Key Results
- Perplexity Reduction: The best language model, incorporating generated CS text (50M words), automatically transcribed data, and machine-translated text, achieved a perplexity of 155.8 on the test set, a substantial improvement over the baseline (231.6).
- Word Error Rate (WER) Reduction:
- The baseline ASR system (trained on 11.5 hours of CS data) had a total WER of 37.8%.
- With automatic annotation of in-domain data, the WER dropped to 32.7%.
- Adding monolingual Dutch data further reduced the WER to 27.9%.
- Including Flemish data and applying data augmentation brought the WER down to 27.1%.
- The best performing system, incorporating enriched CS LMs and RNN-LM rescoring, achieved a total WER of 23.5%. This represents a significant improvement over the baseline.
Limitations and Future Work
- Flemish Data Contribution Ambiguity: While adding Flemish data improved performance, the authors note it's difficult to definitively conclude whether the improvement is due to accent robustness or simply the increased data volume.
- Future Work: The paper suggests further exploration into the impact of Flemish data and potentially other language varieties. The success of synthetic data generation also opens avenues for more sophisticated text generation techniques.
Summary updated