Automatic Transcription of Handwritten Old Occitan Language

Esteban Garces Arias, Vallari Pai, Matthias Schöffel, Christian Heumann, Matthias AßenmacherEMNLP 2023 · DOI

Introduction

Old Occitan, the language of the medieval Troubadours, was once a dominant literary and administrative tongue across Southern Europe from the 11th to the 16th centuries. Today, it represents a significant cultural heritage but remains a "low-resource" language in the digital humanities. A massive archive of approximately 600,000 handwritten dictionary cards exists, documenting word variants and their standardized lemmas. However, 93% of these cards remain untranscribed, hindering large-scale linguistic research.

The Challenge of Medieval Handwriting

Digitizing these records presents unique hurdles. Unlike modern printed text, medieval handwriting is highly non-standardized. It features irregular layouts, unique diacritics, and various abbreviations. General-purpose Optical Character Recognition (OCR) engines, such as Google Cloud Vision, often struggle with these nuances, as they are primarily optimized for modern languages and standardized fonts.

Examples of handwritten dictionary cards showing the mapping from graphical variants to standardized lemmas.

Methodology: A Transformer-Based Approach

To address these challenges, researchers developed a specialized Handwritten Text Recognition (HTR) system using a Transformer-based encoder-decoder architecture.

Architecture

  • Encoder (Swin Transformer): The system employs a Swin Transformer as the vision backbone. Unlike standard Vision Transformers that process images in fixed-size patches, the Swin Transformer uses a hierarchical structure with "shifted windows." This allows the model to capture both fine-grained strokes and the broader global structure of handwritten characters across different scales.
  • Decoder (BERT): A BERT-based architecture serves as the text decoder. The researchers found that training this decoder from scratch on the specific Old Occitan character set yielded better results than fine-tuning a model pre-trained on modern languages.
  • Tokenizer: A byte-level Byte Pair Encoding (BPE) tokenizer was used to handle the text at a granular level, which is crucial given the non-standardized nature of the vocabulary.

Training Strategy

The model underwent a two-stage training process:

  1. Pre-training: The model was first trained on 900,000 synthetic images. These images were generated by sampling characters from an extended EMNIST dataset (including special characters like the arrow symbol and cedilla) and arranging them according to a massive synthetic corpus of 82 million word pairs.
  2. Fine-tuning: The pre-trained model was then refined using a smaller set of approximately 33,000 real, human-labeled images from the dictionary cards.

Datasets and Data Augmentation

The primary dataset consists of 41,634 human-labeled handwritten cards. To overcome the scarcity of labeled data, the authors leveraged massive synthetic data generation. They compiled a synthetic corpus from existing Old Occitan literary and administrative texts.

A handwritten card example showing the mapping 'SERVOYSA → CERVEZA' with additional phonetic or orthographic notes.

The ablation studies highlighted that synthetic data was the most critical factor in performance. Increasing the synthetic training set from 180,000 to 900,000 images provided the single largest boost in accuracy.

Results and Benchmarking

The Swin + BERT model achieved state-of-the-art performance:

  • Label Accuracy: 96.5%
  • Weighted Character Error Rate (CER): 0.005 (0.5%)

In comparative benchmarks, the specialized model significantly outperformed commercial and open-source alternatives. While the proposed system reached 96.5% accuracy, Google Cloud Vision achieved only 46.5%, and open-source tools like Tesseract and PaddleOCR lagged further behind at 17.7% and 28.8%, respectively.

Limitations and Future Directions

Despite its success, the model faces challenges with complex layouts. Accuracy drops from 97.1% for single-line cards to 71.4% for cards containing three lines of text or irregular marginalia. Additionally, very rare symbols and specific French diacritics remain difficult for the system to recognize consistently. Future work will focus on improving layout analysis and adapting the model to handle cursive or lowercase manuscripts more effectively.

The researchers have released their models and datasets on Hugging Face, providing a valuable resource for the digitization of other historical and low-resource languages.

Summary updated

BibTeX
@inproceedings{arias2023automatic,
  author = {Esteban Garces Arias and Vallari Pai and Matthias Schöffel and Christian Heumann and Matthias Aßenmacher},
  title = {{Automatic Transcription of Handwritten Old Occitan Language}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.953},
}
Figure