Adapting Whisper for Regional Dialects: Enhancing Public Services for Vulnerable Populations in the United Kingdom

Melissa Torgbi, Andrew Clayman, Jordan J. Speight, Harish Tayyar MadabushiCOLING 2025

TL;DR

This paper investigates how well OpenAI's Whisper large-v3 model handles regional UK English dialects in public service contexts, finding it struggles compared to its performance on standard English. Fine-tuning Whisper on specific regional data significantly improves its accuracy for those dialects, suggesting a path towards more inclusive AI for vulnerable populations.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are increasingly integrated into public services, but they often exhibit biases against non-standard accents and dialects. This can lead to miscommunication and disadvantage individuals, particularly vulnerable populations who may have regional accents. While large, general-purpose ASR models like Whisper are powerful, their performance on underrepresented dialects, especially in specific real-world domains like public services, remains underexplored. This paper aims to bridge this gap by evaluating Whisper's effectiveness on two distinct UK regional dialects within the public service sector and exploring fine-tuning as a solution.

Methods

The researchers employed a multi-stage approach:

  1. Out-of-the-box Evaluation: They first assessed the performance of the pre-trained Whisper large-v3 model on novel datasets representing two UK regional dialects (North East Scotland and South East Scotland) from public service organizations.
  2. Fine-tuning: They then fine-tuned Whisper models specifically on data from each of these regions. This involved training the models further on the collected regional data to adapt them to the specific accents and linguistic nuances.
  3. Manual Error Analysis: To gain a deeper understanding beyond quantitative metrics like Word Error Rate (WER), they conducted a manual inspection of model errors. This helped identify the types of mistakes made, the impact of transcription style differences, and the effectiveness of WER as an evaluation metric.
  4. Post-processing: Several automated post-processing steps were applied to human transcripts to normalize variations in spelling, date/time formatting, and spacing, aiming to create a fairer comparison and isolate recognition errors from transcription style differences.

Datasets

  • Baseline Dataset: Open-source Multi-speaker Corpora of the English Accents in the British Isles dataset (Demirsahin et al., 2020). This dataset was used for initial comparisons but lacked the specific domain relevance of the newly collected data.
  • Novel Datasets:
    • NESAC (North East Scotland Advice Charity): Call data from a charity providing free legal advice and housing support in North East Scotland.
  • SESHA (South East Scotland Housing Association): Call data from a housing association in South East Scotland. These datasets were manually annotated with accent labels and transcribed by human annotators to serve as ground truth. A subset was used for fine-tuning, and the remainder for testing.

Key Insights and Ideas

  • Whisper's Struggle with Regional Dialects: The state-of-the-art Whisper model, despite its broad training, shows a significantly higher Word Error Rate (WER) on these specific UK regional dialects compared to its performance on a baseline dataset. This highlights that "general" robustness doesn't always translate to specific, underrepresented accents.
  • Fine-tuning is Highly Effective: Fine-tuning Whisper on data from a specific region dramatically improves its performance on that region's dialect. The fine-tuned models consistently outperformed the base Whisper model on their respective test sets.
  • Dialect Transferability is Possible (with caveats): While fine-tuned models performed best on their own training data, they also showed some improved performance on data from the other region, suggesting a degree of dialect transferability within parts of the UK.
  • WER Limitations: The manual error analysis revealed that WER can be misleading. Transcription style differences (e.g., spelling variations, date formats, spacing) can artificially inflate WER, masking genuine improvements in accent recognition.
  • Fine-tuning Trade-offs: While fine-tuning enhances accent recognition, it can sometimes lead to a loss of contextual understanding or introduce "contextual bias," where the model becomes overly sensitive to phonetic patterns at the expense of meaning.

Key Results

  • Whisper large-v3 achieved a WER of 3.64% on the baseline data but saw higher WERs on the NESAC (33.6%) and SESHA (22.2%) test datasets.
  • Fine-tuning significantly reduced WER:
  • The NESAC fine-tuned model achieved a WER of 24.0% on NESAC test data.
  • The SESHA fine-tuned model achieved a WER of 17.3% on SESHA test data.
  • The fine-tuned models generally outperformed the base Whisper model on both test datasets, with the best performance achieved when the fine-tuning data matched the test data.
  • Manual analysis identified specific error types, including phonetic discrimination issues, contextual biases (e.g., misinterpreting "mutually" as "neutrally"), and date/time formatting discrepancies.

Limitations and Future Work

  • Limited Scope: The study focused on only two specific UK regional dialects. Future work should expand to include a wider range of UK accents to assess transferability more broadly.
  • Data Sensitivity: The use of sensitive data from public service organizations prevented public sharing, which is a common challenge in this domain. Future research could explore methods that avoid or minimize the use of confidential data.
  • Annotation Bias: While efforts were made, annotation bias in human transcripts might still persist, emphasizing the continued need for manual analysis.
  • Language Focus: The study was limited to English. Investigating fine-tuning for other languages with regional variations would be valuable.
  • Future Directions: The authors suggest further investigation into the transferability of fine-tuned models across regions and exploring approaches that mitigate the need for sensitive data. They also highlight the need for a balanced fine-tuning approach to avoid contextual bias.

Summary updated

BibTeX
@inproceedings{torgbi2025adapting,
  author = {Melissa Torgbi and Andrew Clayman and Jordan J. Speight and Harish Tayyar Madabushi},
  title = {{Adapting Whisper for Regional Dialects: Enhancing Public Services for Vulnerable Populations in the United Kingdom}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
}
Figure