TL;DR
The ATHAAR dataset introduces the first manually annotated corpus specifically for medieval scientific Arabic manuscripts, bridging a critical gap in Handwritten Text Recognition (HTR). The study reveals a surprising performance gap: general-purpose Vision-Language Models (VLMs) like Gemini 1.5 Flash significantly outperform models specialized for Arabic or historical HTR, though all models still struggle with the complex ligatures and calligraphic flourishes of ancient scientific texts.
What Problem Does This Paper Address?
While HTR has made massive strides in Latin and Chinese, medieval Arabic scientific manuscripts remain a "final frontier." These documents—primary sources for the history of philosophy, astronomy, and mathematics—present a unique cocktail of challenges:
- Cursive Complexity: Arabic script is inherently cursive, featuring complex ligatures and variable diacritization.
- Domain Specificity: Scientific texts are dense with numerals (often in varying systems) and geometrical terminology that standard literary or religious HTR datasets don't cover.
- Calligraphic Flourishes: Scribes frequently used tatweel (elongation of characters) for justification, which can confuse modern OCR systems.
- Data Scarcity: Before this paper, there were no publicly available, rigorously annotated datasets for scientific Arabic manuscripts that included multi-witness variations (different copies of the same text).
Methods
The researchers developed the ATHAAR dataset using a three-tiered approach to test model transferability:
- Tier 1: The same chapter across 13 different scribes (testing robustness to handwriting styles).
- Tier 2: Different chapters within the same treatise (testing robustness to content changes).
- Tier 3: Entirely different scientific treatises (testing out-of-domain generalization).
They established the first formal annotation guidelines for this domain, rooted in "diplomatic editing"—a method from textual criticism that requires transcribing exactly what the scribe wrote without modernizing spelling or correcting errors.
For evaluation, they benchmarked three classes of models:
- General Multimodal: Gemini 1.5 Flash.
- Historical Specialized: CHURRO-3B (fine-tuned on 46 languages).
- Arabic Specialized: Qari-OCR-2B (fine-tuned on modern and classical Arabic).
Datasets
The paper introduces ATHAAR, which consists of 3,893 lines across 169 pages, manually transcribed at the line level. It primarily features al-Qānūn al-Mas‘ūdī, an astronomical treatise by the polymath al-Bīrūnī (c. 1035), alongside other works by figures like Ibn Sīnā and al-Tūsī.
The authors compared ATHAAR against existing Arabic HTR datasets, highlighting that ATHAAR is the only one to provide both a formal annotation guide and measured inter-annotator agreement.
Key Insights and Ideas
- The VLM Dominance: General-purpose VLMs (Gemini) are currently more effective at "reasoning" through messy historical scripts than models specifically trained for HTR. Specialized models often suffered from "runaway generation," producing thousands of characters in response to a single elongated stroke (tatweel).
- The "Tatweel" Trap: Scribal elongation is a major hurdle. In one instance, a single tatweel stroke led Gemini to hallucinate 4,037 characters for a line that only contained 11. This underscores the absolute necessity of text normalization before scoring HTR performance.
- Scribe > Script: The difficulty of transcription depends more on the individual scribe's hand and the physical condition of the manuscript than the broad script category (e.g., Naskh vs. Nasta’liq).
- Few-Shot Stability: While few-shot prompting (k=3) helped Gemini slightly, it actually destabilized specialized models like Qari-OCR, causing their error rates to skyrocket.
Key Results
The quality of the ground truth was validated with a high inter-annotator agreement, achieving a Character Error Rate (CER) of only 2.6%.
In the model evaluation, Gemini 1.5 Flash was the clear winner across all transfer scenarios, maintaining a CER between 16.3% and 21.7%. In contrast, specialized models like CHURRO and Qari-OCR often saw CERs exceeding 70%, failing to produce usable transcriptions for these specific scientific contexts.
Limitations and Future Work
The authors acknowledge that ATHAAR is currently limited to a specific period and a handful of authors. The next logical steps include:
- Fine-tuning: The current study only looked at zero-shot and few-shot performance; fine-tuning open-weight VLMs on ATHAAR is a high-priority next step.
- Explicit Modeling: Developing HTR architectures that explicitly handle calligraphic features like tatweel and complex ligatures.
- Expanded Tasks: Using the dataset's polygon coordinates to improve line segmentation and script identification tasks.
Summary updated