Abstract
Historical text normalization is a vital preprocessing step for digital humanities, yet it is often bottlenecked by the high cost of manual labeling and the ambiguity of archaic spellings. This paper introduces a semi-supervised generative framework that combines a "noisy channel" model with contextual information from modern language models. By utilizing unlabeled historical data through Expectation-Maximization, the proposed approach achieves state-of-the-art results in low-resource settings, reaching 88% of fully supervised performance with zero labeled training pairs.
Introduction: The Problem of Historical Text Normalization
Historical text normalization involves mapping archaic word forms to their modern counterparts. This process allows modern NLP tools—such as part-of-speech taggers and parsers—to be applied to historical corpora, which is essential for digital humanities research.
The task faces two major challenges:
- Data Scarcity: Most existing methods require thousands of manually normalized word pairs, which are expensive to produce.
- Contextual Ambiguity: Historical spellings are often ambiguous. For example, the historical German word desz could normalize to das, des, or dessen depending on its grammatical role.
Methodology: Contextualized Generative Modeling
The researchers propose a generative model based on the "noisy channel" principle. Instead of directly predicting the modern word, they model the joint probability $p(x, y)$ of the historical sequence ($x$) and modern sequence ($y$): $$p(x, y) \approx p(y) \prod p(x_i | y_i)$$
Model Components
- Language Model ($p(y)$): Captures the syntax of modern text using either a Neural HMM or an RNN Language Model trained on modern corpora like Wikipedia.
- Neural Channel ($p(x|y)$): A character-level neural transducer that models how modern words are transformed into historical spellings.
- Semi-supervised Learning: The model uses Expectation-Maximization (EM) to learn from unlabeled historical text. It iteratively "guesses" normalizations and updates its parameters to improve those guesses.
- Candidate Generation: To handle large vocabularies, the system uses heuristic search and a "direct model" transducer to propose modern word candidates.
Datasets
The approach was evaluated across eight historical datasets from four language families:
- Germanic: English, German, Icelandic, and Swedish.
- Romance: Portuguese and Spanish.
- Slavic: Slovene.
- Finno-Ugric: Hungarian.
The datasets vary in age (14th–19th centuries) and genre, including religious texts and scientific writings.
Key Results and Insights
The proposed model demonstrates significant advantages in realistic, low-resource scenarios.
- Low-Resource Efficiency: With only 500 labeled tokens, the model achieved an average error reduction of 19% compared to previous state-of-the-art systems.
- Zero-Shot Performance: In a purely unsupervised setup, the system reached 88.4% of the accuracy of fully supervised models. This highlights the power of using modern language models as a source of indirect supervision.
- Handling Ambiguity: In German, the contextual model reduced errors on ambiguous tokens by 38% compared to non-contextual baselines, proving that context is a powerful supervisor.
- The "Direct Model" Bootstrap: Using a "direct model" to generate candidates helped the system handle words not present in the modern training dictionary or those with extreme spelling variations.
Limitations and Future Work
Despite its success, the model has limitations:
- Tokenization: It assumes pre-tokenized text and struggles with many-to-one alignments (e.g., aller handt $\rightarrow$ allerhand).
- Domain Mismatch: Modern Wikipedia text may not perfectly represent the language of specialized historical domains like 15th-century theology.
- Future Directions: The authors suggest moving toward segmental recurrent neural networks (SRNNs) to handle tokenization and normalization jointly.
Conclusion
This work shows that semi-supervised, contextualized models can effectively bridge the gap between historical and modern text without requiring massive labeled datasets. By leveraging the "goldmine" of unlabeled historical data and the rich context of modern language models, the framework provides a robust solution for low-resource historical text processing.
Summary updated