Beyond Orthography: Automatic Recovery of Short Vowels and Dialectal Sounds in Arabic

Yassine El Kheir, Hamdy Mubarak, Ahmed Ali, Shammur Absar ChowdhuryACL 2024 · DOI

Introduction

Arabic speech technology faces a significant hurdle known as diglossia—the gap between Modern Standard Arabic (MSA), used in formal writing, and various regional dialects (DA) used in daily speech. Standard Arabic orthography is an "abjad," primarily recording consonants while omitting short vowels (diacritics). Furthermore, regional dialects introduce unique phonemes (e.g., the Egyptian 'g') and borrowed sounds (e.g., 'p') that lack standard written representations. Consequently, models trained on standard text struggle to capture the full phonetic reality of spoken Arabic. This paper introduces the Dialectal Sound and Vowelization Recovery (DSVR) framework to automatically reconstruct these hidden sounds from audio.

Methodology: The DSVR Framework

The DSVR framework utilizes self-supervised learning and vector quantization to bridge the gap between acoustic signals and verbatim transcriptions. The pipeline operates in three stages:

  1. Feature Extraction: Raw audio is processed through XLS-R, a 1-billion parameter multilingual self-supervised learning (SSL) model, to generate high-dimensional continuous representations.
  2. Vector Quantization (VQ): Continuous features are mapped to a discrete "codebook" of $k=256$ labels using k-means clustering. This process creates a learned phonetic alphabet that abstracts away speaker-specific noise.
  3. The DVR Model: A transformer-based model predicts verbatim transcriptions from these codes. Two variants were evaluated: DVR Discrete, which uses only quantized codes, and DVR Joint, which integrates both discrete codes and continuous SSL embeddings.
The DSVR framework pipeline, illustrating the flow from raw speech to recovered phonetic units.

Experimental Setup

The researchers introduced ArabVoice15, a new 5-hour test set spanning 15 Arab countries with phonetically accurate verbatim transcriptions. Other datasets included Ar:CVR (enhanced Common Voice), EgyAlj (Egyptian Arabic news), and English datasets (LibriSpeech/TIMIT) to assist in learning borrowed phonemes like /p/ and /v/.

Key Results and Insights

The DSVR model achieved a 7% improvement in Character Error Rate (CER) on the ArabVoice15 test set compared to a strong SSL-only baseline.

Comparison of the baseline architecture with the proposed DVR Discrete and DVR Joint models.

A critical finding was the architectural preference of different sound types:

  • Short Vowels: These were best recovered by the DVR Joint model, indicating that vowels require the fine-grained information preserved in continuous embeddings.
  • Dialectal Consonants: These were effectively captured by the DVR Discrete model, suggesting that quantization acts as a beneficial "denoising" mechanism for categorical phonetic units.

Human perceptual tests validated that the discovered clusters (with an optimal size of 256) represent real-world dialectal nuances, often capturing sounds that exist between canonical phonemes.

Conclusion and Future Work

DSVR demonstrates that even with limited annotated data (as little as 90 minutes), it is possible to recover the rich phonetic detail of Arabic dialects. While the current study is limited by the size of ArabVoice15 and potential transcriber bias, future work will focus on scaling to more dialects and applying these recovered units to tasks like dialect-specific Text-to-Speech (TTS).

Summary updated

BibTeX
@inproceedings{kheir2024beyond,
  author = {Yassine El Kheir and Hamdy Mubarak and Ahmed Ali and Shammur Absar Chowdhury},
  title = {{Beyond Orthography: Automatic Recovery of Short Vowels and Dialectal Sounds in Arabic}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.711},
}
Figure