An Unsupervised Model of Orthographic Variation for Historical Document Transcription

Dan Garrette, Hannah Alpert-AbramsNAACL 2016

TL;DR

This paper introduces a novel unsupervised OCR model that simultaneously generates both "diplomatic" (preserving original spelling) and "normalized" (modernized spelling) transcriptions of historical documents. By learning a probabilistic mapping between modern language models and historical orthography, the system significantly reduces transcription errors for both tasks, outperforming previous state-of-the-art methods.

What Problem Does This Paper Address?

Transcribing historical documents presents a significant challenge due to the inconsistent and archaic orthography (spelling, abbreviations, etc.) used in the past. Standard Optical Character Recognition (OCR) tools often aim for "diplomatic" transcriptions, preserving these variations, which are valuable for historical analysis but difficult for modern NLP tasks. Conversely, "normalized" transcriptions, which modernize the text, are easier to read and process but are typically achieved through a separate, error-prone post-processing step. This paper addresses the need for a unified approach that can produce both types of transcriptions accurately and efficiently, without relying on manually created rules or extensive linguistic expertise.

Methods

The authors build upon the "Ocular" OCR model, a generative approach that models the physical printing process to handle challenges like unknown fonts and image degradation. Their key innovation is the introduction of a new "glyph layer" that decouples the language model (LM) character from the rendered character.

Here's how it works:

  • Standard Ocular: The LM generates a character (e.g., 'a'), and this character is directly rendered.
  • New Model: The LM generates a character (e.g., 'a'), but then a glyph character (e.g., 'a' with a tilde for shorthand, or 'a' with an accent) is chosen probabilistically based on the LM character and the language. This chosen glyph character is then rendered.

This probabilistic mapping between LM characters and glyph characters is learned unsupervised using an Expectation-Maximization (EM) algorithm. The model learns to associate common orthographic variations with their modern equivalents, effectively creating a dynamic, data-driven normalization process integrated directly into the transcription pipeline. They also incorporate constraints to ensure the learned substitutions are linguistically plausible (e.g., letter-to-letter, diacritics only on lowercase letters).

Datasets

The authors evaluate their model on the "Primeros Libros" corpus, which contains multilingual (Spanish, Latin, Nahuatl) books from the 1500s. They used existing gold diplomatic transcriptions and introduced two new monolingual Spanish books annotated with both diplomatic and normalized transcriptions. For training language models (LMs), they utilized texts from Project Gutenberg and a private collection for Nahuatl, ensuring these texts followed modernization standards.

Key Insights and Ideas

  • Unsupervised Learning for Orthographic Variation: The ability to learn orthographic variations and their probabilistic mappings without manual rules or supervision is a significant advancement. This makes the approach more scalable and adaptable to different languages and historical periods.
  • Joint Diplomatic and Normalized Transcription: Producing both types of transcriptions simultaneously is a novel and practical contribution. This eliminates the need for a separate, error-prone normalization pipeline.
  • Bridging Historical Text and Modern NLP: By normalizing historical texts during transcription, the model makes historical documents more amenable to modern NLP techniques, potentially improving downstream tasks like sentiment analysis or topic modeling on historical corpora.
  • Metadata Generation: The learned substitution rules and their probabilities can serve as valuable metadata about orthographic variation in historical printing.

Key Results

  • Diplomatic Transcription: Achieved a 35% relative error reduction compared to the baseline Ocular system. This performance is comparable to a previous approach that used hand-constructed rules.
  • Normalized Transcription: Achieved a 46% relative error reduction over the baseline Ocular system and a 28% reduction over the hand-built rule-based approach.
  • The model successfully learned and applied common substitution rules, as demonstrated in Table 4.

Limitations and Future Work

  • Accent Ambiguity: The model sometimes struggles to distinguish between dots and accents, leading to incorrect accent placement (e.g., adding an accent where it's not present in modern Spanish).
  • Idiomatic Shorthands: The model has difficulty with highly specialized or idiomatic shorthands that lack clear modern equivalents.
  • Future Work: The authors suggest that their joint model approach could be extended to other NLP tasks, such as part-of-speech tagging or parsing, by training these models jointly with the orthographic normalization. This could overcome limitations of pipeline approaches.

Summary updated

BibTeX
@inproceedings{garrette2016unsupervised,
  author = {Dan Garrette and Hannah Alpert-Abrams},
  title = {{An Unsupervised Model of Orthographic Variation for Historical Document Transcription}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
}
Figure