Introduction
While Natural Language Processing (NLP) has made significant strides in Question Answering (QA), most benchmarks rely on "gold-standard" text—perfectly transcribed, grammatically correct strings. In practice, millions of users interact with these systems via voice interfaces. This real-world usage introduces two critical challenges: Automatic Speech Recognition (ASR) noise and dialectal diversity. Errors in transcription propagate into QA models, and systems optimized for standard language varieties often fail users with diverse dialects.
The SD-QA benchmark is designed to evaluate QA systems in these messy, real-world conditions. By augmenting the TyDi-QA dataset with over 68,000 audio prompts across 24 dialects and 5 languages, this research quantifies how interface noise and speaker demographics impact system utility and fairness.
The SD-QA Dataset
The primary contribution of this work is the Spoken Dialectal QA (SD-QA) dataset. Building on the multilingual TyDi-QA benchmark, SD-QA covers five languages: Arabic, Bengali, English, Kiswahili, and Korean.
The dataset includes recordings from 255 native speakers, capturing 24 distinct dialects. This includes 11 varieties of English (e.g., from South Africa, the Philippines, and Nigeria) and 7 varieties of Arabic (e.g., from Algeria to Saudi Arabia). This breadth allows for a rigorous evaluation of how well models generalize across global linguistic variations.
Methodology
The researchers employed a pipeline approach to benchmark current state-of-the-art systems:
- Audio Collection: Native speakers recorded questions from the TyDi-QA dataset.
- ASR Phase: Audio was transcribed using the Google Speech-to-Text API, comparing "general" models against "localized" models (e.g.,
en-USvs.en-IN). - QA Phase: The resulting transcripts were fed into a multilingual BERT-based QA model.
- Fairness Evaluation: A "Generalized Entropy Index" was used to measure how equitably system benefits (accuracy) are distributed across different dialectal and demographic groups.
Key Findings
The ASR Bottleneck
QA performance is heavily dependent on ASR quality. For instance, in Kiswahili, where the Word Error Rate (WER) was approximately 43%, QA performance dropped by 14 percentage points compared to gold-standard text. Interestingly, while localized ASR models often improved transcription accuracy, these gains did not always translate to better downstream QA results, suggesting that QA models may be resilient to certain types of phonetic noise but vulnerable to others.
The Fairness Gap
The study revealed significant "fairness gaps" between user populations. Systems were found to be notably less fair to Kiswahili and English speakers compared to Bengali speakers, primarily due to the high performance variance between different dialects within those languages. In Arabic, the challenge was compounded by "diglossia"—the mismatch between local spoken dialects and the Modern Standard Arabic (MSA) used in the underlying QA context.
Conclusion and Future Work
The SD-QA benchmark highlights that evaluating on clean text overestimates the performance of QA systems in production. Future research should address the limitations of this study, such as the use of read speech versus spontaneous speech and the reliance on location as a proxy for dialect. A promising direction is the development of End-to-End Spoken QA models that bypass the transcription step entirely, potentially mitigating the propagation of ASR errors.
Summary updated