TL;DR
This paper introduces AfriSpeech-200, the first large-scale, open-source dataset of Pan-African accented English speech, comprising 200 hours from over 2,400 speakers across 120 accents. It addresses the critical gap in ASR for African languages and accents, particularly in the clinical domain, and provides pre-trained models that significantly improve performance on these underrepresented speech patterns.
What Problem Does This Paper Address?
The paper tackles the significant challenge of poor Automatic Speech Recognition (ASR) performance for African accents, especially in specialized domains like clinical settings. While ASR technology is mature and widely used in developed nations, its application in Africa is hampered by a lack of diverse, representative datasets. This leads to a "performance gap," where existing ASR systems struggle with the nuances of African English accents, potentially hindering the adoption of productivity tools in critical sectors like healthcare. The low doctor-to-patient ratio in many African countries exacerbates this problem, as efficient clinical documentation tools could alleviate clinician burden. Furthermore, existing research often exhibits racial bias and performs poorly on non-native English speakers, highlighting the need for datasets that capture the true linguistic diversity of Africa.
Methods
The authors developed a multi-pronged approach to create the AfriSpeech-200 dataset and evaluate ASR models.
- Dataset Creation: They employed a combination of web scraping (news sites, health-focused websites), leveraging existing biomedical datasets (PubMed, NCBI disease), and augmenting these with African-centric entities (names, cities) and "Africanized" template sentences. This templating approach involved masking proper nouns and replacing them with African entities, repeating each template 200 times to increase data volume.
- Audio Recording: A web-based application was developed for crowd-sourced audio collection, allowing for tracking of completion, demographics, and quality control. Recorders were instructed to use their natural accents and pronounce punctuation marks.
- Quality Control: A rigorous quality control process was implemented, involving human reviewers who up-voted or down-voted clips, provided feedback, and a tiered system for admitting and retaining recorders based on clip quality. Problematic sentences were identified and blacklisted.
- Model Fine-tuning: Two top-performing open-source ASR architectures, wav2vec-large-xlsr-53 and whisper-medium, were selected for fine-tuning on the AfriSpeech dataset. This fine-tuning was performed across general, clinical, and combined domains to assess domain adaptation capabilities.
- Vocabulary Expansion: A crucial step involved defining an alphanumeric vocabulary that included numbers and punctuation, recognizing their importance in clinical contexts, unlike many pre-trained models that focus solely on Latin alphabets.
Datasets
- AfriSpeech-200: The primary contribution is this newly introduced dataset, featuring 200.70 hours of Pan-African accented English speech from 2,463 unique speakers, covering 120 indigenous accents across 13 countries. It includes 67,577 speech-transcript pairs, categorized into clinical and general domains.
- Existing Datasets (for pre-training and benchmarking):
- Wikitext-103
- PubMed
- NCBI disease
- Common Voice
- SautiDB
- Librispeech (used for benchmarking and comparison)
Key Insights and Ideas
- The "Performance Gap" is Real and Significant: The study clearly demonstrates that standard ASR models, even state-of-the-art ones trained on large datasets like Librispeech, perform drastically worse on African accents. This highlights a critical need for inclusive dataset development.
- Domain Adaptation is Nuanced: While fine-tuning on the AfriSpeech dataset improves performance across the board, the paper shows that cross-domain adaptation (e.g., general to clinical) yields benefits, but the trend can reverse if not carefully managed. This suggests that simply adding more data isn't always the solution; the type and diversity of data are paramount.
- Clinical ASR Requires Specialized Vocabulary: The importance of including numbers and punctuation in the ASR vocabulary for clinical applications is emphasized. Models that fail to accurately transcribe these elements, common in medical texts, will have significantly higher Word Error Rates (WER).
- Commercial ASR APIs are Not Universally Effective: Despite their global presence, major commercial ASR systems show considerable performance degradation on African accents, underscoring the limitations of their training data and the need for localized solutions.
- Diversity is Key to Generalizability: Fine-tuning with a diverse dataset like AfriSpeech significantly improves zero-shot performance on unseen accents, demonstrating that broader linguistic coverage in training data leads to more robust and generalizable ASR systems.
Key Results
- AfriSpeech-200 Dataset: Provides 200.70 hours of speech, 67,577 clips, from 2,463 speakers across 120 accents in 13 countries.
- Fine-tuning Improvements: Fine-tuning XLSR-53 on AfriSpeech-general resulted in a 53.4% relative improvement, while fine-tuning on clinical data yielded a 52.6% improvement. Whisper-medium saw similar gains (32.6% and 32.1% respectively).
- Accent-Level Performance: Fine-tuned models achieved an average relative improvement of 26.7% over open-source models and 36.5% over commercial models on AfriSpeech accents.
- Zero-Shot Performance: Fine-tuned Whisper models showed a 44.4% relative performance improvement on out-of-distribution (OOD) accents compared to baselines.
- Commercial ASR Performance: Commercial ASR APIs showed significant performance drops on African accents compared to their performance on majority African accents. For example, on Swahili and Yoruba, performance lagged significantly.
Limitations and Future Work
- Limited Clinical Subdomains: The dataset includes a variety of clinical text, but several specialties are not represented, meaning ASR performance might vary across different clinical areas.
- Read Speech vs. Conversational Speech: The current dataset consists of read speech based on prompts. ASR models trained on this may underperform on spontaneous or conversational speech without further augmentation.
- North-African Accents Excluded: The work focuses on sub-Saharan accents, and performance may not generalize to North African regions due to their distinct nature.
- Self-Reported Accents: Recorders self-reported their native tongues, which can be imprecise and lead to overlaps (e.g., Zulu and IsiZulu). A normalized accent field is provided to mitigate this.
- Medical Abbreviations Inconsistency: Variations in how abbreviations are pronounced (e.g., "Breast CA" vs. "Breast Cancer") can lead to transcript errors and penalize models.
- Integrating ASR in Healthcare: Challenges like privacy concerns, security risks, and the potential for unethical ASR vendor practices remain significant hurdles for widespread clinical adoption.
- Future Work: The authors hope future users will investigate AfriSpeech's utility and quality for their specific languages and accents, and further explore the impact of accent representation in pre-training data. They also suggest further investigation into the performance on specific African languages and the potential for improving ASR for spontaneous speech.
Summary updated