Corpus Development of Kiswahili Speech Recognition Test and Evaluation sets, Preemptively Mitigating Demographic Bias Through Collaboration with Linguists

Kathleen Siminyu, Kibibi Mohamed Amran, Abdulrahman Ndegwa Karatu, Mnata Resani, Mwimbi Makobo Junior, Rebecca Ryakitimbo, Britone MwasaruACL 2022

TL;DR

This paper details the creation of a more inclusive Kiswahili speech recognition dataset by collaborating with linguists to capture dialectal and demographic diversity. The project addresses the known biases in speech recognition systems (gender, age, accent) by proactively building a dataset that represents the nuances of Kiswahili speakers, aiming to improve model performance across a wider range of users.

What Problem Does This Paper Address?

Speech recognition technologies are becoming ubiquitous, yet they often exhibit significant biases. Research has consistently shown that these systems perform better for men than women and for younger individuals compared to older ones. This disparity is particularly concerning in low-resource language contexts, such as many spoken in the Global South. When developing datasets for these languages, there's a risk of replicating existing biases and failing to capture the full spectrum of linguistic variation. This paper tackles this challenge by focusing on Kiswahili, a widely spoken language in East Africa, and aims to build a more representative dataset that accounts for not only different dialects but also the subtle variations in pronunciation and speech patterns influenced by factors like age, gender, and regional background.

Methods

The core of this paper's methodology lies in a unique, collaborative approach involving linguists and language experts. The process began with preliminary preparation to understand existing knowledge about Kiswahili variations and to define desired outputs for the dataset. This involved identifying dominant dialects, accents, and linguistic features.

A key methodological innovation was the Linguists Engagement phase. Instead of solely relying on computational methods, the researchers actively recruited and collaborated with linguists who possessed deep knowledge of Kiswahili. These linguists were chosen based on their familiarity with the language, geographical spread (Kenya, Tanzania, DRC), gender diversity, and personal contributions to the Kiswahili community.

The methodology then involved a series of interactive discussions where linguists shared their expertise. These discussions were transcribed and used to inform the data collection process. The team also conducted fieldwork, encouraging collaboration with native speakers to create textual data (word lists and sentences) and audio recordings. Various engagement methods were employed, including relying on linguists' subjective experiences, analyzing existing video and audio content, conducting focus groups, and engaging with people through communication platforms.

Datasets

The primary dataset introduced in this paper is a Kiswahili Speech Recognition dataset specifically designed to mitigate demographic bias. This dataset is a subset of the broader Common Voice dataset and focuses on 10 specific dialects and variants of Kiswahili. The selection of these dialects was guided by criteria such as the number of speakers, national representation (Kenya, Tanzania, DRC), and linguistic diversity across major dialectal categories (Northern, Central, Southern).

The paper also references the Common Voice dataset as the overarching project within which this work is situated. Common Voice is described as a massively multilingual speech corpus developed for Automatic Speech Recognition (ASR) purposes, aiming for diversity and inclusivity.

Key Insights and Ideas

  • The Critical Role of Linguists: The paper strongly emphasizes that building inclusive language technology requires deep linguistic expertise. Linguists are essential for understanding the nuanced differences in language use that go beyond simple dialectal classifications.
  • Socio-linguistic Nuances Matter: The research highlights that factors like age, gender, accent, and even the influence of other languages spoken in a region (code-switching) significantly impact speech patterns. These subtle variations are crucial for building truly equitable ASR systems.
  • Standardization vs. Diversity: The paper touches upon the historical context of Kiswahili standardization, noting how the creation of "Standard Kiswahili" (Kiswahili Sanifu) was largely driven by colonial authorities and may have marginalized other dialects. This historical perspective underscores the importance of actively seeking out and preserving linguistic diversity.
  • Beyond Dialects: Variants are Key: The research recognizes that linguistic variation extends beyond formal dialects to include regional, social, and contextual differences in language use, termed "variants." Capturing these variants is crucial for comprehensive representation.
  • Multidisciplinarity is Essential: The project's success underscores the importance of multidisciplinary teams in developing datasets and NLP technologies. Combining computational expertise with linguistic knowledge leads to more robust and equitable outcomes.

Key Results

The paper details the creation of textual and audio resources for 10 Kiswahili dialects/variants. Table 2 shows the number of words/phrases and sentences created for each, totaling 10,669 words/phrases and 2,172 sentences. The authors state that these resources have been uploaded to GitHub and will be made available in future releases of the Common Voice dataset. The primary goal is to use these subsets for quantitatively evaluating ASR model performance across different demographic groups and to provide resources for developers to fine-tune models for specific local contexts.

Limitations and Future Work

Limitations:

  • Subset Size: The developed subsets are intentionally smaller than the overall Kiswahili dataset, serving as a focused evaluation tool rather than a comprehensive replacement.
  • Limited Scope of Variants: Due to time and resource constraints, the study focused on 5 main variant clusters and did not fully incorporate variants involving code-switching (e.g., Sheng) or those mixing Kiswahili with French in the DRC.

Future Work:

  • Expansion of Text Resources: The authors plan to expand text resources, aiming for 5,000 unique sentences per dialect/variant.
  • Voice Data Collection: Following text expansion, efforts will focus on collecting voice data for these expanded resources.
  • Replication and Broader Coverage: The paper encourages others to replicate these efforts for dialects and variants not covered in this study.
  • Inclusion of Code-Switching Variants: Future work should prioritize the inclusion of code-switching variants to achieve greater linguistic equity.

Summary updated

BibTeX
@inproceedings{siminyu2022corpus,
  author = {Kathleen Siminyu and Kibibi Mohamed Amran and Abdulrahman Ndegwa Karatu and Mnata Resani and Mwimbi Makobo Junior and Rebecca Ryakitimbo and Britone Mwasaru},
  title = {{Corpus Development of Kiswahili Speech Recognition Test and Evaluation sets, Preemptively Mitigating Demographic Bias Through Collaboration with Linguists}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure