Uncertainty-driven Partial Diacritization for Arabic Text

Humaid Alblooshi, Artem Shelmanov, Hanan Aldarmaki2025 · DOI

Abstract

We propose an uncertainty-based approach to Partial Diacritization (PD) for Arabic text. We evaluate three uncertainty metrics for this task: Softmax Response, BALD via MC-dropout, and Mahalanobis Distance. We further introduce a lightweight Confident Error Regularizer to improve model calibration. Our preliminary exploration illustrates possible ways to use uncertainty estimation for selectively retaining or discarding diacritics in Arabic text with an analysis of performance in terms of correlation with diacritic error rates. For instance, the model can be used to detect words with high diacritic error rates which tend to have higher uncertainty scores at inference time. On the Tashkeela dataset, the method maintains low diacritic error rates while reducing the amount of visible diacritics on the text by up to 50% with thresholding-based retention.

Introduction

Arabic script utilizes diacritics, known as Tashkeel, to represent short vowels and phonetic nuances. While essential for disambiguation and for non-native speakers, fully diacritized text can be visually dense and potentially impede reading speed for fluent readers. The objective of Partial Diacritization (PD) is to provide a sufficient number of marks to ensure clarity without overwhelming the reader. This research explores using neural network "uncertainty" to automatically determine which diacritics are most critical to retain.

Methodology

The study utilizes the Character-based Arabic Tashkeel Transformer (CATT) as the base model. To implement uncertainty-driven PD, three metrics were evaluated:

  1. Softmax Response (SR): A baseline measuring the maximum probability of the predicted class.
  2. BALD (Bayesian Active Learning by Disagreement): Calculated using Monte Carlo dropout to measure consistency across multiple stochastic passes.
  3. Mahalanobis Distance (MD): A distance-based metric measuring how far an input's internal representation is from the training distribution.

To enhance the reliability of these metrics, a Confident Error Regularizer (CER) was introduced. This loss function penalizes high-confidence incorrect predictions, thereby improving model calibration and the alignment between uncertainty scores and actual error likelihood.

Global Error Rate vs. Coverage Trade-off across different CER regularization strengths (Lambda).

Experimental Setup

The models were trained and evaluated on the Tashkeela Corpus, a comprehensive dataset of classical and modern Arabic text. Additionally, the ArVoice corpus was used as an out-of-domain test set to evaluate the robustness of the uncertainty metrics across different linguistic styles.

Results and Discussion

The experiments revealed that uncertainty metrics can effectively guide diacritic retention strategies. Two primary approaches were identified: keeping high-confidence marks to ensure accuracy, or keeping low-confidence marks to assist readers with the most ambiguous tokens.

A key finding was that the computationally simple Softmax Response (SR) performed comparably to the more complex BALD method, while both significantly outperformed Mahalanobis Distance (MD) in identifying erroneous diacritics.

Refined Diacritic Error Rate vs. Percentage of Diacritics Kept for SR, BALD, and MD metrics.

As shown in the figures, the system can achieve a 50% reduction in visible diacritics while maintaining a very low Diacritic Error Rate (DER). Specifically, by filtering out the 20% most uncertain diacritics, the model achieved an absolute reduction in error rates to approximately 2.5% DER. While the CER improved calibration, it resulted in a slight trade-off in raw accuracy, as the model became more "cautious" in its predictions.

Conclusion

This work demonstrates that model uncertainty is a viable and efficient tool for automated partial diacritization. Future research will focus on human-in-the-loop studies to measure the impact of these PD strategies on reading comprehension and speed. Additionally, the authors plan to integrate these uncertainty-aware models into downstream tasks like Machine Translation and Text-to-Speech to mitigate the impact of linguistic ambiguity.

Summary updated

Abstract

We propose an uncertainty-based approach to Partial Diacritization (PD) for Arabic text.We evaluate three uncertainty metrics for this task: Softmax Response, BALD via MC-dropout, and Mahalanobis Distance.We further introduce a lightweight Confident Error Regularizer to improve model calibration.Our preliminary exploration illustrates possible ways to use uncertainty estimation for selectively retaining or discarding diacritics in Arabic text with an analysis of performance in terms of correlation with diacritic error rates.For instance, the model can be used to detect words with high diacritic error rates which tend to have higher uncertainty scores at inference time.On the Tashkeela dataset, the method maintains low diacritic error rates while reducing the amount of visible diacritics on the text by up to 50% with thresholding-based retention.
BibTeX
@misc{alblooshi2025uncertainty,
  author = {Humaid Alblooshi and Artem Shelmanov and Hanan Aldarmaki},
  title = {{Uncertainty-driven Partial Diacritization for Arabic Text}},
  year = {2025},
  doi = {10.18653/v1/2025.uncertainlp-main.1},
}
Figure