TL;DR
This paper explores semi-supervised learning to improve automatic speech recognition (ASR) for code-switched speech in South Africa, where multiple languages are mixed within sentences. They found that leveraging automatically transcribed speech significantly reduces word error rates for both bilingual and a novel five-lingual ASR system, though the five-lingual system shows a bias towards the more resourced English language.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems typically perform best when trained on large amounts of transcribed speech. However, in many real-world scenarios, especially in multilingual regions like South Africa, speech often involves "code-switching" – the seamless alternation between two or more languages within a single conversation or even a single sentence. This poses a significant challenge for ASR systems, as they struggle to model the complex linguistic patterns and acoustic variations. Furthermore, many of the African languages spoken in South Africa are "under-resourced," meaning they lack the extensive text and audio data needed to train robust ASR models. This paper tackles the problem of improving ASR for code-switched speech involving English and four South African Bantu languages (isiZulu, isiXhosa, Setswana, and Sesotho) by investigating semi-supervised training techniques to make better use of limited transcribed data and a larger pool of untranscribed speech.
Methods
The core of this research lies in semi-supervised training. The authors employed a two-phase approach:
- Automatic Transcription: They used existing, well-trained ASR systems to automatically transcribe a large amount of untranscribed, manually segmented code-switched speech. Two types of transcription systems were used:
- Bilingual Transcribers (AutoTB): Four separate systems, each trained for a specific English-Bantu language pair (English-isiZulu, English-isiXhosa, English-Setswana, English-Sesotho).
- Unified Five-Lingual Transcriber (AutoTF): A single system trained on all five languages simultaneously.
- Acoustic Model Retraining: The automatically transcribed speech was then combined with the original manually transcribed data to retrain the acoustic models. This process aims to leverage the vast amount of untranscribed data to improve the models' performance.
The acoustic models themselves were based on Time Delay Neural Networks (TDNN-F), a state-of-the-art architecture for ASR, and an enhanced version incorporating Convolutional Neural Networks (CNN-TDNN-F). The CNN layers were hypothesized to improve performance by capturing local acoustic patterns.
Datasets
The primary dataset used is a South African soap opera corpus. This corpus contains:
- Manually transcribed data: Approximately 11 hours of code-switched speech, balanced across four language pairs (English-isiZulu, English-isiXhosa, English-Setswana, English-Sesotho). This was augmented with about 9 hours of monolingual English soap opera speech.
- Manually segmented, untranscribed data: An additional 11.05 hours of speech from 127 speakers, which was used for the automatic transcription phase of semi-supervised training.
The authors also mention using monolingual NCHLT speech data for the five target languages as part of the initial training of their transcription systems.
Key Insights and Ideas
- Semi-supervised learning is effective for under-resourced code-switched ASR: Even with limited manually transcribed data, using automatically transcribed speech significantly reduces word error rates. This is crucial for languages where creating large transcribed datasets is prohibitively expensive.
- Bilingual vs. Unified Five-Lingual Systems: While both approaches benefit from semi-supervised training, the unified five-lingual system offers flexibility by handling more than two languages simultaneously. However, it exhibits a bias towards English, likely due to English being a common language across all pairs and thus having more training data.
- CNNs benefit bilingual, not unified systems: The CNN-TDNN-F architecture showed significant improvements for the bilingual systems, but not for the unified five-lingual system. This suggests that the added complexity of the five-lingual task might outweigh the benefits of CNNs in this specific configuration.
- Language bias in unified systems: The unified five-lingual system's tendency to favor English is a direct consequence of the data distribution. This highlights a common challenge in multilingual ASR: balancing performance across all languages when one is significantly more resourced.
- Code-switching points are critical: The accuracy at the exact points where languages switch is a key metric. The bilingual systems performed better at these specific points, likely due to less ambiguity compared to the unified system.
Key Results
- Bilingual TDNN-F: One iteration of semi-supervised training (using AutoTB) led to an absolute WER reduction of 2.68% relative to the baseline.
- Bilingual CNN-TDNN-F: Further improved performance with an additional absolute WER reduction of 2.43% over the bilingual TDNN-F with semi-supervised training.
- Unified Five-Lingual CNN-TDNN-F: Achieved an error rate that was "almost as good as that attained by the bilingual systems," despite the inherent complexity of decoding five languages.
- Language-specific WERs (Table 5): The unified five-lingual system showed higher WERs for Bantu languages compared to English, confirming the bias. The bilingual systems generally had better performance on Bantu languages.
Limitations and Future Work
- English Bias: The unified five-lingual system's bias towards English is a significant limitation, impacting performance on the under-resourced Bantu languages.
- CNNs for Unified Systems: The lack of improvement with CNN-TDNN-F for the unified system suggests further architectural exploration might be needed.
- Future Work: The authors suggest incorporating improved automatic segmentation, as well as speaker and language diarization, into the pipeline. This would further enhance the ability to process and leverage untranscribed data, potentially mitigating the language bias and improving overall performance.
Summary updated