A Large-Scale Comparison of Historical Text Normalization Systems

Marcel BollmannNAACL 2019 · DOI

Abstract

Historical text normalization—the process of mapping archaic spellings to modern forms—is a critical preprocessing step for applying modern NLP tools to historical documents. This study provides a large-scale evaluation of rule-based, statistical, and neural normalization systems across eight languages. The results demonstrate that while neural models are increasingly popular, character-based Statistical Machine Translation (SMT) remains the most effective solution for the small-to-medium datasets common in historical linguistics.

Introduction

Natural Language Processing (NLP) tools, such as part-of-speech taggers and parsers, are typically trained on modern, standardized text. When applied to historical documents, their performance often degrades significantly due to the high degree of orthographic variation. Before the 19th century, spelling was often inconsistent, with a single modern word having dozens of historical variants.

Historical spelling variants of the word "their" mapping to a single modern form.

Normalization serves as a bridge, converting these variants into modern "normal forms." Despite its importance, the field has lacked a standardized comparison across different languages and data scales. This paper addresses this gap by evaluating leading methodologies on a diverse set of historical corpora.

Methodology

The study compares four primary systems representing different technological paradigms:

  1. Norma (Rule-based/Hybrid): A flexible tool combining lookup lists, replacement rules, and weighted edit distance. It is designed for interpretability and low-resource scenarios.
  2. cSMTiser (Statistical): A character-based Statistical Machine Translation (CSMT) approach that treats words as sequences of characters to be "translated" into modern spelling.
  3. NMT (Neural): Two Neural Machine Translation architectures: a "shallow" single-layer LSTM and a "deep" six-layer RNN model.

A key focus of the evaluation is the learning curve of these models—how their performance scales as the amount of training data increases from 100 to 50,000 tokens.

Datasets

The evaluation spans 10 datasets across eight languages: English, German, Hungarian, Icelandic, Spanish, Portuguese, Slovene, and Swedish. These datasets represent various genres (religious texts, letters, science) and time periods (14th to 19th centuries). They also vary in difficulty; for instance, the Hungarian dataset features a high divergence from modern forms (17.5% identity), whereas the English dataset is more standardized (75.3% identity).

The Impact of Data Quantity

The most significant finding involves the "crossover points" where different technologies become viable.

Learning curves for English and Hungarian datasets showing accuracy vs. training data size.
  • Low-Resource Dominance: For extremely small datasets (under 500 tokens), rule-based methods like Norma are superior.
  • SMT Efficiency: SMT models (cSMTiser) scale rapidly, outperforming rule-based methods once approximately 1,000 tokens are available.
  • NMT Data Hunger: Neural models exhibit the steepest learning curves but generally require more data than is available in many historical corpora to surpass SMT.

Key Results and Insights

  • SMT as the "Sweet Spot": Character-based SMT achieved the best or near-best accuracy (often 90–95%) across nearly all tested datasets.
  • Hybrid Strategies: The paper recommends a pipeline approach: use simple dictionary lookups for previously seen words and reserve complex SMT/NMT models for unseen (out-of-vocabulary) tokens.
  • Model Depth in NMT: In neural architectures, depth appears more critical than specific hyperparameter tuning; the "deep" NMT model consistently outperformed the "shallow" version.
  • Monolingual Boost: Integrating a language model trained on modern data into the SMT pipeline (cSMTiser+LM) provides a consistent performance gain by favoring linguistically plausible outputs.

Limitations and Future Directions

The current state-of-the-art models operate at the token level, ignoring surrounding context. This leads to errors in cases of homography, where a historical spelling could map to multiple modern words depending on the sentence. Future research should focus on context-aware normalization, integrating sentence-level information into character-based models to resolve these ambiguities.

Summary updated

BibTeX
@inproceedings{bollmann2019large,
  author = {Marcel Bollmann},
  title = {{A Large-Scale Comparison of Historical Text Normalization Systems}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1389},
}
Figure