A Lattice-Based HMM Approach to Aruz Scansion of Ottoman Poetry

Mehmet Emre TiryakiarXiv 2026 · arXiv

TL;DR

Researchers have developed a new automated system for scanning Ottoman Divan poetry that works directly with modern Turkish spelling. By combining a spelling restorer with a lattice-based Hidden Markov Model (HMM), the system achieves 77.6% full-line accuracy, significantly outperforming rule-based scanners and state-of-the-art large language models like Claude 3 Opus.

What Problem Does This Paper Address?

Ottoman Divan poetry is written in aruz, a quantitative meter where the rhythm is determined by the "weight" (length) of syllables rather than stress. This system was originally designed for the Arabic script, which explicitly records long vowels and specific consonants crucial for the meter.

However, since Turkey's 1928 alphabet reform, these poems are primarily read and typed in modern Latin-script Turkish. This modern spelling is "lossy"—it omits the diacritics that signify syllable weight. For a researcher or student, finding the correct meter and "scanning" a line (aligning syllables to the meter) is slow, error-prone, and requires specialized knowledge of poetic "flaws" (kusur) where poets intentionally deviate from the rules. Existing automated tools either require perfectly transcribed scholarly editions or struggle with the inherent ambiguity of plain text.

Methods

The authors propose a three-part generative model that treats scansion as a joint search problem:

  1. Spelling Restorer: For every plain word, a trained model proposes up to five possible scholarly spellings (with diacritics) and their probabilities.
  2. Lattice Construction: The system builds a graph (lattice) representing every possible way to read a line. This includes different spellings, choices for "linking" (liaison between words), and medd (overlong syllables). The lattice represents multiple readings of a single poetic foot.

  3. Meter HMM: A Hidden Markov Model represents the rhythmic templates of various meters. It uses "match" states for meter positions, "insert" states for extra syllables, and "skip" states for missing ones. The HMM structure allows the model to learn the probability of different poetic "flaws."

Figure 2: The lattice of the foot Kamgir ü. Nodes are the points between syllables and each arc is one syllable with its label; a path from node 0 to node 4 is one reading of the foot. kamgir has two candidate spellings, with the restorer’s probability r of each (its remaining candidates are not drawn), and k¯am and g¯ır each give two alternatives, as described in the text.
Figure 3: The meter HMM for the first foot of the poem’s meter, whose first position may be long or short (x). There is a match state per position, an insert state after each, and a skip over each. Below the states, the reading of Figure 2 as the labels each position emits. The rates are learned by EM from the printed meters alone.

The entire system is trained using Expectation-Maximization (EM). Crucially, it doesn't need manually labeled scansion data; it learns the rates of poetic flaws and spelling preferences simply by looking at poems and their known overall meters.

Datasets

  • LATOC Corpus: 52,000 lines from 13 critical editions of Divan poets (15th–18th century).
  • Annotated Test Set: 500 lines manually annotated for scansion (using an ensemble of LLMs followed by human expert auditing).
  • Robustness Benchmark: A newly created dataset that simulates real-world "noisy" text from web copies and OCR (Optical Character Recognition) errors, based on measured deviation rates in the wild.

Key Insights and Ideas

  • Joint Decoding is Essential: You can't fix the spelling first and then scan the meter. The meter helps you decide the spelling, and the spelling helps you find the meter. Doing this simultaneously in a lattice is what makes the model robust.
  • Unsupervised Learning of Poetic Style: The model successfully learned the frequency of poetic flaws (like imâle, where a short vowel is read long) purely from data. These learned rates match the qualitative descriptions found in traditional philology handbooks.
  • Robustness to "Real-World" Text: By including insert and skip states in the HMM, the model can handle typos, OCR errors, and misprints that would break a rigid rule-based system.

Key Results

The model was evaluated on Task 1 (identifying the meter) and Task 2 (full scansion alignment).

  • Meter Identification: The system identified the correct meter for 99.9% of poems and 98.2% of individual lines.
  • Scansion Accuracy: On plain Turkish text, the model achieved 77.6% full-line accuracy.
  • For comparison, a rule-based scanner managed only 26.8%.
  • Claude 3 Opus (few-shot) reached 63.6%.
  • A standard neural BiLSTM tagger reached 55.0%.
  • Ablation Study: Removing the HMM or the spelling restorer's probabilities caused performance to plummet (by over 40 points), proving that the integration of these components is the "secret sauce."

Limitations and Future Work

The current system is limited to an inventory of the 14 most common meters, which covers the vast majority of Divan poetry but excludes rarer forms. The authors also note that while the model is robust, performance still drops on very noisy OCR data, suggesting a need for even better integration with OCR error correction.

The code and a web-based version of the tool have been released to assist researchers in Turkish literature.

Summary updated

Abstract

Ottoman Divan poetry is composed in aruz, a quantitative meter based on the weights of syllables. The meter is sensitive to specific characters recorded in the original writing in the Arabic script. This work is concerned with automatic meter finding and scansion alignment for Ottoman Divan poetry, especially on modern Turkish spelling without the special diacritics that scholarly transliterations of Ottoman preserve. We define two tasks on this input, first identifying the meter used in the poem, and second aligning the meter's weights with the syllabification of the poem, identifying the flaws and freedoms the poet has taken along the way. We build a corpus of 52,000 lines split by poet, with scansion annotations for 500 test lines, and a robustness benchmark generated at the deviation rates measured on real web and OCR copies. Our model is a hidden Markov model over the positions of each meter, decoded over a lattice of every candidate path, and trained by EM from the printed meter, without scansion labels. On a held-out set it scans 77.6% of the test lines fully correctly, against 26.8% for the available rule-based scanner, 55.0% for a neural tagger trained with the same supervision and 63.6% for Claude Opus 5.5 few-shot. The code is open-sourced and the tool is released on the web.
BibTeX
@misc{tiryaki2026lattice,
  author = {Mehmet Emre Tiryaki},
  title = {{A Lattice-Based HMM Approach to Aruz Scansion of Ottoman Poetry}},
  year = {2026},
  eprint = {2611.12691},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12691},
}
Figure