Introduction
Historical text normalization—the process of mapping archaic spellings (e.g., sayed) to modern equivalents (said)—is a critical preprocessing step for applying modern Natural Language Processing (NLP) tools to historical corpora. While neural models have become the standard for this task, this paper critiques current evaluation methodologies. The authors identify three primary issues: a lack of simple baselines, a failure to distinguish between "seen" and "unseen" words (the "seen word" bias), and a gap between intrinsic accuracy and extrinsic performance on downstream tasks like Part-of-Speech (POS) tagging.
Methods and Datasets
The study evaluates three distinct approaches across five languages (English, German, Hungarian, Icelandic, and Swedish):
- Naïve Baseline (Memorization): A lookup table mapping historical words to their most frequent training-set modern equivalent. If a word is not found, it remains unchanged.
- Soft Attention Model: A standard Bi-LSTM encoder-decoder architecture with a soft attention mechanism.
- Hard Monotonic Attention Model: A model adapted from morphological re-inflection that assumes linear character alignment, theoretically better suited for spelling tasks.
The evaluation utilizes diverse historical corpora, including the Parsed Corpus of Early English Correspondence (PCEEC) for extrinsic POS tagging analysis.
Key Insights: The Power of Baselines and the Challenge of Generalization
A central finding is that for words encountered during training, the naïve memorization baseline is remarkably robust. Neural models often "overthink" these cases, leading to errors where a simple lookup would suffice. In English, for instance, the baseline achieved 96.9% accuracy on seen tokens, outperforming the Soft Attention model's 93.7%.
The true value of neural architectures lies in their ability to generalize to "unseen" tokens. The Hard Attention model consistently outperformed other methods in this regard. As shown in Figure 1, while the percentage of unseen tokens decreases with more data, the accuracy on these tokens is where sophisticated models demonstrate their worth.
Interestingly, the Soft Attention model's performance on seen words sometimes decreased as training data grew, likely due to underfitting larger datasets with default hyperparameters.
Extrinsic Evaluation: POS Tagging
The ultimate goal of normalization is often to improve downstream NLP tasks. The authors found that while normalization significantly boosts POS tagging accuracy (from ~71% to ~77%), the choice of a complex neural model over a simple baseline provides diminishing returns as the volume of training data increases.
As illustrated in Figure 2, neural models show a statistically significant advantage only when training data is scarce (5k–10k items). At larger scales, the performance gap between the sophisticated models and the naïve baseline effectively closes, suggesting that for many practical applications, simpler methods may be sufficient.
Conclusion
The authors advocate for a shift in how historical normalization systems are reported. Researchers should distinguish between performance on seen and unseen tokens and prioritize extrinsic evaluations. While neural models—particularly those with hard monotonic attention—offer superior generalization for rare or unseen words, their practical advantage for downstream tasks may be less pronounced than intrinsic metrics suggest.
Summary updated