Supervised Grapheme-to-Phoneme Conversion of Orthographic Schwas in Hindi and Punjabi

Aryaman Arora, Luke Gessler, Nathan Schneider

Abstract

Converting written text to speech (Grapheme-to-Phoneme conversion) in Indo-Aryan languages like Hindi and Punjabi is complicated by "schwa deletion." While the Devanagari script implies an inherent schwa vowel (\textit{ə}) for every consonant, spoken forms frequently omit them. This paper presents the first statistical classifier for this task, utilizing gradient-boosting decision trees (XGBoost) to achieve state-of-the-art accuracy (98%) using only orthographic context.

Introduction

In the Devanagari abugida script, consonants are assumed to be followed by an inherent schwa vowel unless another vowel diacritic is present. However, in spoken Hindi and Punjabi, these schwas are often deleted based on complex phonological and prosodic rules. For instance, a word written as pe-pə-rə might be pronounced [pepər], where the final schwa is dropped while the medial one is retained.

Comparison of Devanagari orthography and phonemic representation showing schwa deletion.

Accurately predicting these deletions is critical for high-quality Text-to-Speech (TTS) systems. Previous attempts relied on rigid, hand-crafted linguistic rules that often failed to account for the morphological complexity of these languages or the influence of loanwords from Persian, Arabic, and English.

Methodology and Data

The researchers framed schwa deletion as a supervised binary classification problem. For every orthographic schwa, the model must decide whether it is retained or deleted.

Models and Features

The study compared three machine learning approaches: Logistic Regression, Multi-layer Perceptrons (Neural Networks), and XGBoost. The primary features used were a simple "sliding window" of surrounding characters. Through testing, a window of five characters to the left and five to the right was found to provide optimal context. While the team also tested explicit phonological features (e.g., voicing or place of articulation), the models performed remarkably well using orthography alone.

Datasets

To train and evaluate these models, the authors compiled cleaned datasets from historical dictionaries:

  • Hindi: McGregor (1993) and Bahri (1989) dictionaries.
  • Punjabi: Singh (1895) dictionary.

A linear-time force-alignment algorithm was employed to map written characters to spoken phonemes, providing the ground-truth labels for training. These datasets have been released publicly to support future NLP research in Indo-Aryan languages.

Experimental Results

The XGBoost model emerged as the superior approach, significantly outperforming traditional rule-based systems and other statistical models.

Performance metrics for Hindi and Punjabi across different classification models.

In Hindi, the XGBoost model achieved 98.00% per-schwa accuracy and 97.78% word-level accuracy on the McGregor dataset. This represents a substantial improvement over the transliteration logic used by Wiktionary (94.18%). The approach also proved robust for Punjabi, reaching 94.66% accuracy.

Beyond raw performance, qualitative analysis revealed that the decision trees were implicitly "rediscovering" linguistic rules. For example, the model learned to penalize deletion when the preceding syllable was "heavy," a known principle in Hindi prosody.

Discussion and Future Work

Despite the high accuracy, certain challenges remain. Most errors occurred with loanwords from Sanskrit, Persian, or Arabic, which often deviate from native phonological patterns. Additionally, the current binary model cannot capture "weakened" schwas—vowels that are neither fully deleted nor fully voiced.

Future research may focus on integrating morphological analyzers to handle compound words better, as schwa deletion rules often reset at morpheme boundaries. Explicitly providing prosodic information, such as syllable stress and moraic weight, could also further refine the model's predictive power.

Summary updated

BibTeX
@inproceedings{arora2020supervised,
  author = {Aryaman Arora and Luke Gessler and Nathan Schneider},
  title = {{Supervised Grapheme-to-Phoneme Conversion of Orthographic Schwas in Hindi and Punjabi}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=mFMxy_PK9_},
}
Figure