MD3: The Multi-Dialect Dataset of Dialogues

Jacob Eisenstein, Vinodkumar Prabhakaran, Clara Rivera, Dorottya Demszky, Devyani Sharma

Introduction

Most contemporary speech and language technologies are "dialect-blind," having been trained primarily on a narrow subset of standard American or British English. This narrow focus creates a significant performance gap for the billions of individuals who speak "World Englishes"—varieties such as Indian or Nigerian English that possess distinct phonological, syntactic, and discourse patterns.

The Multi-Dialect Dataset of Dialogues (MD3) addresses a critical bottleneck in the development of inclusive AI: the lack of high-quality, naturalistic data. Existing dialect datasets often fall into two extremes: they are either scripted, which eliminates natural variation, or entirely unconstrained, making it difficult to measure whether a model accurately understands a speaker's intent. MD3 occupies the "Goldilocks zone" by utilizing structured information-sharing tasks that elicit spontaneous speech while providing a clear ground truth for communication goals.

Methodology: The Guessing Game Framework

To capture authentic conversational dynamics, the researchers employed a collaborative "guessing game" framework. This setup ensures that speakers use their natural vocabulary and grammar to achieve specific objectives. The dataset consists of two primary task types:

  1. Image-Guessing: A "describer" views a specific image (e.g., a particular breed of dog) and must describe it so that a "guesser" can identify it from a lineup of twelve similar images. Prompts were sourced from established datasets such as FoodX-251, CalTech-UCSD Birds, and Stanford Dogs.
  2. Word-Guessing: Similar to the game Taboo, speakers must describe a target word (e.g., "Science Fiction") without using specific forbidden "taboo" words (e.g., "Future," "Travel"). Prompts for this task were drawn from the Tabouid dataset.

Conversations were recorded remotely via a web interface and subsequently transcribed by professional orthographic transcribers from the same geographic regions as the speakers to ensure linguistic accuracy.

Dataset Composition

MD3 features over 20 hours of audio and 200,000 tokens across 3,689 games. The corpus focuses on three specific locales:

  • en-IN: Native Telugu speakers in Hyderabad, India.
  • en-NG: Native Yoruba speakers in Nigeria.
  • en-US: Native English speakers in the Western United States.

Despite the dialectal differences, the task-oriented nature of the dataset proved successful; participants maintained high win rates for the image games, exceeding 86% across all three regions.

Key Insights and Benchmarking

The "Whisper" Performance Gap

The researchers benchmarked OpenAI’s Whisper, a state-of-the-art automatic speech recognition (ASR) model, against the MD3 dataset. The results highlighted a stark disparity in performance. Whisper’s Word Error Rate (WER) was significantly higher for Nigerian English (38.6) and Indian English (33.2) compared to American English (22.3). This confirms that even models marketed as "robust" struggle with the nuances of global English varieties.

Capturing Unique Dialectal Features

MD3 successfully captured specific linguistic features often ignored in standard Natural Language Processing (NLP) research:

  • Syntactic Variation: The "stative progressive" (using the -ing suffix for states, such as "it is having a chain on its neck") appeared frequently in the Indian and Nigerian cohorts but was entirely absent in the US cohort.
  • Discourse Markers: The dataset revealed distinct usage patterns for markers of focus and confirmation. Indian English speakers used the word "done" as a confirmation marker (e.g., "Okay, okay. Done.") 219 times, whereas US speakers used it only once in the same context. Similarly, the use of "only" as a focus marker (e.g., "chocolate cake only") was unique to the Indian English group.

Limitations and Future Work

While MD3 represents a significant step forward, the authors acknowledge several limitations. The dataset currently represents only one primary first-language (L1) background per region (Telugu for India and Yoruba for Nigeria), which does not capture the full linguistic diversity of these nations. Furthermore, the prompts—often drawn from Wikipedia and Western-centric datasets—may have introduced a bias that favored the speed of US speakers.

Future iterations of the project aim to expand the dataset to include Spanish and Portuguese dialects. Additionally, the researchers plan to use the inherent "intent" data within the games to test Natural Language Understanding (NLU) systems, moving beyond simple speech-to-text benchmarks to evaluate how well AI understands the meaning behind dialectal speech.

The MD3 dataset is publicly available for research purposes on Kaggle.

Summary updated

BibTeX
@inproceedings{eisenstein2023md,
  author = {Jacob Eisenstein and Vinodkumar Prabhakaran and Clara Rivera and Dorottya Demszky and Devyani Sharma},
  title = {{MD3: The Multi-Dialect Dataset of Dialogues}},
  booktitle = {Interspeech},
  year = {2023},
  doi = {10.21437/interspeech.2023-2150},
}
Figure