Homograph Disambiguation through Selective Diacritic Restoration

Sawsan Alqahtani, Hanan Aldarmaki, Mona T. Diab

TL;DR

Fully restoring omitted Arabic diacritics disambiguates homographs but severely fragments vocabulary and degrades downstream NLP model performance due to data sparsity. This paper introduces data-driven, unsupervised methods to identify truly ambiguous words and selectively restore diacritics only to those tokens. Across translation, semantic similarity, and syntactic tagging, selective diacritization strikes an optimal balance between disambiguation and vocabulary sparsity.

What Problem Does This Paper Address?

In languages like Arabic and Hebrew, short vowels and pronunciation marks (diacritics) are almost universally omitted in standard writing. Omitting them multiplies the number of homographs—words that share identical surface spelling but carry different pronunciations and meanings (e.g., the Arabic string ktb can mean "wrote", "books", or "was written").

While fully restoring diacritics theoretically eliminates lexical ambiguity, in practice it explodes vocabulary size, increases out-of-vocabulary (OOV) rates, and introduces data sparsity, degrading downstream machine learning performance. Conversely, leaving text completely undiacritized preserves vocabulary density but maximizes ambiguity. The authors address the challenge of finding the sweet spot: selectively restoring diacritics only to words that genuinely benefit from disambiguation, without inflating vocabulary sparsity.

Methods

The authors frame selective diacritization as a dictionary-lookup problem, creating ambiguity dictionaries (AmbigDict) where word types are labeled as ambiguous or unambiguous:

Raw Corpus ──> Unsupervised Ambiguity Detection ──> AmbigDict (Ambiguous vs. Unambiguous)
                                                            │
Input Text ──> Selective Diacritic Restoration  <───────────┘
                (Full lexical diacritics applied ONLY to ambiguous words)

They introduce two data-driven paradigms to construct these dictionaries:

  1. Undiacritized Text Approaches (AmbigDict-UNDIAC):
  2. MULTI (Baseline): Uses the MADAMIRA morphological analyzer; any word with more than one valid diacritized reading is flagged as ambiguous regardless of context.
  3. SENSE: Applies graph-based word sense induction (Chinese Whispers algorithm) over contextual word embeddings. A word occurring in multiple induced sense clusters is labeled ambiguous.
  4. Diacritized Text Approaches (AmbigDict-DIAC):
  5. Clustering (CL): Texts are lexically diacritized (ignoring syntactic inflectional endings). Standard clustering algorithms—Brown clustering (CL-BR), K-means (CL-KM), and Gaussian Mixture Models via EM (CL-EM)—cluster word contexts. If diacritized variants of the same surface form land in different clusters, the word is marked ambiguous.
  6. Translation-based (TR): Leverages bilingual sentence alignment with English via fast-align. If diacritized variants share top-$N$ English translations, they are treated as unambiguous; if they diverge in translation, they are marked ambiguous.

During inference, only words marked ambiguous retain their restored lexical diacritics (via MADAMIRA), while unambiguous words remain undiacritized.

Datasets

  • Pretraining & Ambiguity Mining: ~50M Modern Standard Arabic (MSA) tokens pooled from Gigaword (5th ed.), Arabic Wikipedia (2016 dump), the Corpus of Contemporary Arabic (CCA), and the LDC Arabic Tree Bank (ATB). For the translation-based method (TR), an Arabic–English parallel corpus of ~60M tokens from 53 LDC catalogs was used.
  • Downstream Evaluations:
  • Semantic Textual Similarity (STS): SemEval-2017 Task 1 Arabic benchmark, augmented with sampled sentences.
  • Neural Machine Translation (NMT): IWSLT 2016 WIT3 Arabic–English parallel talk dataset.
  • Part-of-Speech (POS) Tagging: Arabic Tree Bank (ATB parts 1, 2, and 3) annotated with Universal Dependencies v2 tags.

Key Insights and Ideas

  • Context distribution separates true ambiguity from noise: Word variants with nearly identical distributional representations generate redundant vocabulary parameters. True homographs that require disambiguation exhibit divergent distributional contexts or distinct cross-lingual translations.
  • Syntactic vs. semantic task divergence: Full diacritization harms semantic tasks like STS (0.593 Pearson vs. 0.608 for undiacritized) and NMT (26.8 BLEU vs. 27.1 for undiacritized) due to sparsity. Conversely, syntactic tasks like POS tagging benefit from diacritics (98.06% full vs. 97.99% none).
  • Selective restoration captures the best of both worlds: Selective schemes matched or exceeded the top-performing baseline for every downstream task, proving that partial restoration dampens sparsity while injecting critical disambiguating signals.

Key Results

  • Semantic Textual Similarity (STS): The clustering approach CL-EM (0.617) and translation approach TR (0.616) achieved statistically significant improvements over the best baseline (NONE at 0.608).
  • Machine Translation (NMT): TR achieved 27.3 BLEU, significantly outperforming both the undiacritized baseline (27.1) and full diacritization (26.8). On the homograph-only evaluation subset, TR reached 27.6 BLEU (vs. 27.4 and 27.0).
  • POS Tagging: MULTI achieved the highest accuracy (98.11%), outperforming full diacritization (98.06%). It also preserved stronger OOV accuracy (87.51%) compared to full diacritization (87.43%).

Limitations and Future Work

  • Task dependency: No single selection strategy dominated across all tasks; semantic tasks favored translation- and mixture-model-based dictionaries, whereas syntactic tagging favored high-coverage morphological rules (MULTI).
  • Inconclusive linguistic rules: When inspecting abstracted diacritic patterns (e.g., $CaCaC$ vs. $CuCiC$), only ~20 patterns were consistently marked ambiguous across both clustering and translation approaches; the authors found no overarching syntactic or semantic rule that fully accounted for the learned ambiguity assignments.
  • Future directions: Deeper investigation into the abstracted diacritic patterns to construct linguistically grounded, optimal diacritization schemes that generalize across domains without manual annotation.

Summary updated

Abstract

Lexical ambiguity, a challenging phenomenon in all natural languages, is particularly prevalent for languages with diacritics that tend to be omitted in writing, such as Arabic. Omitting diacritics leads to an increase in the number of homographs: different words with the same spelling. Diacritic restoration could theoretically help disambiguate these words, but in practice, the increase in overall sparsity leads to performance degradation in NLP applications. In this paper, we propose approaches for automatically marking a subset of words for diacritic restoration, which leads to selective homograph disambiguation. Compared to full or no diacritic restoration, these approaches yield selectively-diacritized datasets that balance sparsity and lexical disambiguation. We evaluate the various selection strategies extrinsically on several downstream applications: neural machine translation, part-of-speech tagging, and semantic textual similarity. Our experiments on Arabic show promising results, where our devised strategies on selective diacritization lead to a more balanced and consistent performance in downstream applications.
BibTeX
@inproceedings{alqahtani2019homograph,
  author = {Sawsan Alqahtani and Hanan Aldarmaki and Mona T. Diab},
  title = {{Homograph Disambiguation through Selective Diacritic Restoration}},
  booktitle = {WANLP @ ACL},
  year = {2019},
  url = {https://openreview.net/forum?id=TvnnwfuOnl},
}

Loading related content...

Figure