SD-QA: Spoken Dialectal Question Answering for the Real World

Fahim Faisal, Sharlina Keshava, Md Mahfuz Ibn Alam, Antonios Anastasopoulos

Introduction

While Natural Language Processing (NLP) has made significant strides in Question Answering (QA), most benchmarks rely on "gold-standard" text—perfectly transcribed, grammatically correct strings. In practice, millions of users interact with these systems via voice interfaces. This real-world usage introduces two critical challenges: Automatic Speech Recognition (ASR) noise and dialectal diversity. Errors in transcription propagate into QA models, and systems optimized for standard language varieties often fail users with diverse dialects.

The SD-QA benchmark is designed to evaluate QA systems in these messy, real-world conditions. By augmenting the TyDi-QA dataset with over 68,000 audio prompts across 24 dialects and 5 languages, this research quantifies how interface noise and speaker demographics impact system utility and fairness.

The SD-QA Dataset

The primary contribution of this work is the Spoken Dialectal QA (SD-QA) dataset. Building on the multilingual TyDi-QA benchmark, SD-QA covers five languages: Arabic, Bengali, English, Kiswahili, and Korean.

Geographic and dialectal coverage of the SD-QA dataset across five major languages.

The dataset includes recordings from 255 native speakers, capturing 24 distinct dialects. This includes 11 varieties of English (e.g., from South Africa, the Philippines, and Nigeria) and 7 varieties of Arabic (e.g., from Algeria to Saudi Arabia). This breadth allows for a rigorous evaluation of how well models generalize across global linguistic variations.

Methodology

The researchers employed a pipeline approach to benchmark current state-of-the-art systems:

  1. Audio Collection: Native speakers recorded questions from the TyDi-QA dataset.
  2. ASR Phase: Audio was transcribed using the Google Speech-to-Text API, comparing "general" models against "localized" models (e.g., en-US vs. en-IN).
  3. QA Phase: The resulting transcripts were fed into a multilingual BERT-based QA model.
  4. Fairness Evaluation: A "Generalized Entropy Index" was used to measure how equitably system benefits (accuracy) are distributed across different dialectal and demographic groups.
The SD-QA pipeline: from audio input to ASR transcription and final QA answer.

Key Findings

The ASR Bottleneck

QA performance is heavily dependent on ASR quality. For instance, in Kiswahili, where the Word Error Rate (WER) was approximately 43%, QA performance dropped by 14 percentage points compared to gold-standard text. Interestingly, while localized ASR models often improved transcription accuracy, these gains did not always translate to better downstream QA results, suggesting that QA models may be resilient to certain types of phonetic noise but vulnerable to others.

The Fairness Gap

The study revealed significant "fairness gaps" between user populations. Systems were found to be notably less fair to Kiswahili and English speakers compared to Bengali speakers, primarily due to the high performance variance between different dialects within those languages. In Arabic, the challenge was compounded by "diglossia"—the mismatch between local spoken dialects and the Modern Standard Arabic (MSA) used in the underlying QA context.

Conclusion and Future Work

The SD-QA benchmark highlights that evaluating on clean text overestimates the performance of QA systems in production. Future research should address the limitations of this study, such as the use of read speech versus spontaneous speech and the reliance on location as a proxy for dialect. A promising direction is the development of End-to-End Spoken QA models that bypass the transcription step entirely, potentially mitigating the propagation of ASR errors.

Summary updated

Abstract

Question answering (QA) systems are now available through numerous commercial applications for a wide variety of domains, serving millions of users that interact with them via speech interfaces. However, current benchmarks in QA research do not account for the errors that speech recognition models might introduce, nor do they consider the language variations (dialects) of the users. To address this gap, we augment an existing QA dataset to construct a multi-dialect, spoken QA benchmark on five languages (Arabic, Bengali, English, Kiswahili, Korean) with more than 68k audio prompts in 24 dialects from 255 speakers. We provide baseline results showcasing the real-world performance of QA systems and analyze the effect of language variety and other sensitive speaker attributes on downstream performance. Last, we study the fairness of the ASR and QA models with respect to the underlying user populations. The dataset, model outputs, and code for reproducing all our experiments are available: https://github.com/ffaisal93/SD-QA.
BibTeX
@inproceedings{faisal2021sd,
  author = {Fahim Faisal and Sharlina Keshava and Md Mahfuz Ibn Alam and Antonios Anastasopoulos},
  title = {{SD-QA: Spoken Dialectal Question Answering for the Real World}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.281},
}
Figure