This study by Jacob Moore and Paula Lauren from Lawrence Technological University provides a much-needed reality check for Tibetan Optical Character Recognition (OCR). While the field has traditionally relied on Character Error Rate (CER) to measure success, this paper introduces the first public evaluation of syllable- and word-level metrics, revealing that what looks good on a character basis doesn't always translate to usable text for downstream AI tasks.
TL;DR
The researchers introduce syllable-level (SER) and word-level (SWER) error rates for Tibetan OCR, demonstrating that while character-level metrics are generally reliable, they can fail to predict how well a system performs on tasks like translation or named-entity recognition. They find that the syllable—explicitly marked in Tibetan script—is a highly effective "middle ground" metric that avoids the need for complex word segmenters while providing better insight into text quality than characters alone.
What Problem Does This Paper Address?
Tibetan is an abugida script with a unique orthographic challenge: it explicitly marks syllable boundaries (using a small dot called a tsheg), but it does not mark word boundaries.
In most languages, Word Error Rate (WER) is the gold standard for evaluation because it correlates well with human readability and downstream task performance. For Tibetan, researchers have defaulted to Character Error Rate (CER) because computing WER requires a "segmenter"—an additional AI tool to guess where words start and end—which introduces its own errors.
The authors tackle two main questions:
- Does CER actually lead us to the "best" models, or are we being misled by character-level noise?
- How do these different error rates predict the performance of downstream tasks like Part-of-Speech (POS) tagging, Named-Entity Recognition (NER), and Machine Translation (MT)?
Methods
The authors evaluated nine different recognizer configurations across three architecture families:
- CTC Models: Connectionist Temporal Classification models (CRNN and Easter2.0), which are standard for sequence recognition.
- TrOCR: Transformer-based encoder-decoders (Small and Base versions), tested with and without 8-bit and 4-bit quantization.
- Vision-Language Models (VLM): Specifically Qwen3-VL-2B, adapted using 4-bit QLoRA.
They introduced Syllable Error Rate (SER), which treats the text between tsheg marks as the fundamental unit. They also tested three variants of Segmented Word Error Rate (SWER) using different segmenters: a dictionary-based tool (BoTok), a BERT-based classifier, and a Large Language Model (Gemini 2.5 Flash).
To measure real-world utility, they used "preservation scores." They ran downstream tasks (POS, NER, MT) on both the "clean" reference text and the "noisy" OCR output, measuring how much of the original information was preserved.
Datasets
The study utilized four publicly available datasets representing the diverse visual styles of Tibetan text:
- KhyentseWangpo: Modern digital typesetting.
- Karmapa8: Traditional woodblock prints.
- Drutsa: Cursive handwriting.
- Betsug: Calligraphic "book form" handwriting.
The researchers also created mixed-domain combinations to test how well models generalize when trained on multiple styles simultaneously.
Key Insights and Ideas
- The "Whitespace Trap": CTC-based models (like CRNN) often struggle with predicting whitespace correctly. Because CER counts every character edit, a model might get a "bad" CER score simply for missing spaces, even if every Tibetan syllable is perfectly recognized. SER ignores these delimiters, providing a more functional view of the model's accuracy.
- SER is the "Goldilocks" Metric: The authors found that SER ranks systems almost identically to word-level metrics (SWER) but doesn't require a segmenter. This makes it an ideal standard for future Tibetan OCR research.
- Downstream Fragility: Different tasks have different "breaking points." NER is incredibly sensitive; even a low CER can cause entity extraction to fail if the error falls exactly on a name. Translation is more robust, able to "hallucinate" through minor character errors to find the correct meaning.
- Quantization is "Free": For TrOCR models, 8-bit quantization resulted in negligible performance drops, suggesting that researchers can significantly reduce memory usage without sacrificing accuracy.
Key Results
- Top Performer: Surprisingly, the relatively small CRNN model recorded the lowest CER in all four single-domain conditions, though TrOCR-base was highly competitive on typeset text.
- Metric Correlation: CER and SER generally agree on which model is better (Kendall $\tau \geq 0.833$), but they disagreed significantly in 7 out of 64 joint-training comparisons.
- Utility Thresholds: The authors established rough "guidance" levels for what error rates are needed for usable output:
- To achieve 80% POS tag preservation, you need an SER $\leq$ 0.078.
- To achieve a Translation chrF score of 60, you need an SER $\leq$ 0.047.
- NER remains difficult to predict, as errors are often concentrated in the very spans (names/places) that the task cares about.
Limitations and Future Work
The authors acknowledge that their data splits were grouped by page, not by scribe. This means a model might have seen other pages from the same hand during training, potentially inflating performance on the handwriting datasets.
Future work aims to extend this analysis to other scripts that lack word boundaries, such as Dzongkha (Bhutan), Thai, Lao, and Khmer. They also suggest moving from evaluating isolated lines to evaluating the full OCR pipeline, including layout analysis and line segmentation.
Summary updated