Abstract
Hindi grapheme-to-phoneme (G2P) conversion is mostly trivial, with one exception: whether a schwa represented in the orthography is pronounced or unpronounced (deleted). Previous work has attempted to predict schwa deletion in a rule-based fashion using prosodic or phonetic analysis. We present the first statistical schwa deletion classifier for Hindi, which relies solely on the orthography as the input and outperforms previous approaches. We trained our model on a newly-compiled pronunciation lexicon extracted from various online dictionaries. Our best Hindi model achieves state of the art performance, and also achieves good performance on a closely related language, Punjabi, without modification.
The Challenge of Schwa Deletion
In Hindi and Punjabi, the writing systems (like Devanagari) are "abugidas," where each consonant character has an inherent schwa vowel (/ə/) unless another vowel is explicitly marked. However, in spoken language, many of these schwas are "deleted" or silenced based on complex phonological rules. For example, a word written as pe-pə-rə is actually pronounced pe-pər.
Predicting when to delete a schwa is the "final boss" of Hindi G2P conversion. While simple rules like "delete the schwa at the end of a word" work often, internal schwa deletion depends on syllable weight, prosody, and morphology. Previous attempts relied on rigid, hand-coded linguistic rules that often failed to capture the nuance of natural speech or the quirks of loanwords from Persian, Arabic, and English.
Methods and Data
The authors framed schwa deletion as a binary classification problem: for every orthographic schwa, should it be retained or deleted? They compared three machine learning approaches: Logistic Regression, Multi-layer Perceptron (MLP) neural networks, and XGBoost (Gradient-Boosting Decision Trees).
Instead of feeding the model complex linguistic theories, they used a simple sliding window of 5 characters to the left and 5 to the right of the target schwa, represented via one-hot encoding. To create the training data, they compiled and released machine-readable pronunciation datasets from high-quality sources including the McGregor (1993) and Bahri (1989) dictionaries for Hindi, and the Singh (1895) dictionary for Punjabi.
Key Insights
- XGBoost Wins on Interpretability: XGBoost performed the best. The authors found that the decision trees effectively "learned" the same rules linguists have spent decades writing down, such as specific phonotactic constraints regarding consonant clusters.
- Local Context is Sufficient: Surprisingly, the model achieved high accuracy using only a local window of characters. It didn't need explicit information about syllable stress or global word structure to outperform previous systems.
- The "Loanword" Hurdle: The model’s errors were heavily concentrated in loanwords (borrowings from Persian, Arabic, or English) and compound words, which often follow the phonological rules of their source languages rather than native Hindi rules.
- Cross-Language Extensibility: The Hindi-optimized model worked remarkably well on Punjabi with zero modifications, suggesting that the underlying "logic" of schwa deletion is highly conserved across related Indo-Aryan languages.
Key Results
The XGBoost model achieved 98.00% per-schwa accuracy and 97.78% word-level accuracy on the McGregor Hindi dataset. It significantly outperformed the Wiktionary rule-based system, which scored 94.18% word-level accuracy on the same data.
The model also achieved 94.66% accuracy on Punjabi, providing a strong baseline for a language that has seen much less computational research than Hindi.
Limitations and Future Work
Schwa deletion isn't always binary; in some contexts, speakers use a "weakened" schwa that is neither fully present nor fully gone. The current binary classifier cannot capture this phonetic spectrum. Additionally, while the local window works well, adding explicit features for "moraic weight" (syllable heaviness) and morphological boundaries could resolve the remaining 2% of errors. Finally, future work could benefit from more contemporary spoken corpora to account for evolution in colloquial speech since the publication of the source dictionaries.
Summary updated