Syllable- and Word-Level Evaluation of Tibetan OCR

Jacob Moore, Paula LaurenarXiv 2026 · arXiv

This study by Jacob Moore and Paula Lauren from Lawrence Technological University provides a much-needed reality check for Tibetan Optical Character Recognition (OCR). While the field has traditionally relied on Character Error Rate (CER) to measure success, this paper introduces the first public evaluation of syllable- and word-level metrics, revealing that what looks good on a character basis doesn't always translate to usable text for downstream AI tasks.

TL;DR

The researchers introduce syllable-level (SER) and word-level (SWER) error rates for Tibetan OCR, demonstrating that while character-level metrics are generally reliable, they can fail to predict how well a system performs on tasks like translation or named-entity recognition. They find that the syllable—explicitly marked in Tibetan script—is a highly effective "middle ground" metric that avoids the need for complex word segmenters while providing better insight into text quality than characters alone.

What Problem Does This Paper Address?

Tibetan is an abugida script with a unique orthographic challenge: it explicitly marks syllable boundaries (using a small dot called a tsheg), but it does not mark word boundaries.

In most languages, Word Error Rate (WER) is the gold standard for evaluation because it correlates well with human readability and downstream task performance. For Tibetan, researchers have defaulted to Character Error Rate (CER) because computing WER requires a "segmenter"—an additional AI tool to guess where words start and end—which introduces its own errors.

The authors tackle two main questions:

  1. Does CER actually lead us to the "best" models, or are we being misled by character-level noise?
  2. How do these different error rates predict the performance of downstream tasks like Part-of-Speech (POS) tagging, Named-Entity Recognition (NER), and Machine Translation (MT)?

Methods

The authors evaluated nine different recognizer configurations across three architecture families:

  • CTC Models: Connectionist Temporal Classification models (CRNN and Easter2.0), which are standard for sequence recognition.
  • TrOCR: Transformer-based encoder-decoders (Small and Base versions), tested with and without 8-bit and 4-bit quantization.
  • Vision-Language Models (VLM): Specifically Qwen3-VL-2B, adapted using 4-bit QLoRA.

They introduced Syllable Error Rate (SER), which treats the text between tsheg marks as the fundamental unit. They also tested three variants of Segmented Word Error Rate (SWER) using different segmenters: a dictionary-based tool (BoTok), a BERT-based classifier, and a Large Language Model (Gemini 2.5 Flash).

To measure real-world utility, they used "preservation scores." They ran downstream tasks (POS, NER, MT) on both the "clean" reference text and the "noisy" OCR output, measuring how much of the original information was preserved.

Datasets

The study utilized four publicly available datasets representing the diverse visual styles of Tibetan text:

  1. KhyentseWangpo: Modern digital typesetting.
  2. Karmapa8: Traditional woodblock prints.
  3. Drutsa: Cursive handwriting.
  4. Betsug: Calligraphic "book form" handwriting.
Fig. 1. One test line from each script domain, with its reference transcription.

The researchers also created mixed-domain combinations to test how well models generalize when trained on multiple styles simultaneously.

Key Insights and Ideas

  • The "Whitespace Trap": CTC-based models (like CRNN) often struggle with predicting whitespace correctly. Because CER counts every character edit, a model might get a "bad" CER score simply for missing spaces, even if every Tibetan syllable is perfectly recognized. SER ignores these delimiters, providing a more functional view of the model's accuracy.
  • SER is the "Goldilocks" Metric: The authors found that SER ranks systems almost identically to word-level metrics (SWER) but doesn't require a segmenter. This makes it an ideal standard for future Tibetan OCR research.
  • Downstream Fragility: Different tasks have different "breaking points." NER is incredibly sensitive; even a low CER can cause entity extraction to fail if the error falls exactly on a name. Translation is more robust, able to "hallucinate" through minor character errors to find the correct meaning.
  • Quantization is "Free": For TrOCR models, 8-bit quantization resulted in negligible performance drops, suggesting that researchers can significantly reduce memory usage without sacrificing accuracy.

Key Results

  • Top Performer: Surprisingly, the relatively small CRNN model recorded the lowest CER in all four single-domain conditions, though TrOCR-base was highly competitive on typeset text.
  • Metric Correlation: CER and SER generally agree on which model is better (Kendall $\tau \geq 0.833$), but they disagreed significantly in 7 out of 64 joint-training comparisons.
  • Utility Thresholds: The authors established rough "guidance" levels for what error rates are needed for usable output:
  • To achieve 80% POS tag preservation, you need an SER $\leq$ 0.078.
  • To achieve a Translation chrF score of 60, you need an SER $\leq$ 0.047.
  • NER remains difficult to predict, as errors are often concentrated in the very spans (names/places) that the task cares about.

Limitations and Future Work

The authors acknowledge that their data splits were grouped by page, not by scribe. This means a model might have seen other pages from the same hand during training, potentially inflating performance on the handwriting datasets.

Future work aims to extend this analysis to other scripts that lack word boundaries, such as Dzongkha (Bhutan), Thai, Lao, and Khmer. They also suggest moving from evaluating isolated lines to evaluating the full OCR pipeline, including layout analysis and line segmentation.

Summary updated

Abstract

Optical character recognition (OCR) for Tibetan is evaluated predominantly by character error rate (CER). Tibetan marks syllable boundaries but does not mark word boundaries, so a word error rate requires a segmenter. This study evaluates nine recognizer configurations on typeset, woodblock and two handwritten script styles, alone and in combination, and reports syllable- and word-level error rates, to our knowledge the first for Tibetan OCR. The configurations span CTC models, TrOCR encoder-decoders with and without post-training quantization, and a vision-language model, and each is scored under CER, the syllable error rate (SER) and three segmented word error rates (SWER). CER ranks systems largely as the higher-level rates do (Kendall $τ\geq 0.833$), with one pair ordered oppositely. On the effect of joint training, CER and SER record significant changes of opposite sign in 7 of 64 comparisons. SER needs no segmenter and ranks systems as SWER does under two of the three segmenters ($τ\geq 0.889$). The output of each configuration is also tagged for part of speech, annotated for named entities and machine translated, and each result is compared with that obtained on the reference text. As rough guidance, translation chrF of 60 is reached at SER of 0.047 or below and part-of-speech preservation of 0.8 at 0.078 or below, while no error rate separates the systems that reach entity preservation of 0.6 from those that do not.
BibTeX
@misc{moore2026syllable,
  author = {Jacob Moore and Paula Lauren},
  title = {{Syllable- and Word-Level Evaluation of Tibetan OCR}},
  year = {2026},
  eprint = {2611.21303},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21303},
}
Figure