Improving historical spelling normalization with bi-directional LSTMs and multi-task learning

Marcel Bollmann, Anders Søgaard

Abstract

Natural-language processing of historical documents is complicated by the abundance of variant spellings and lack of annotated data. A common approach is to normalize the spelling of historical words to modern forms. We explore the suitability of a deep neural network architecture for this task, particularly a deep bi-LSTM network applied on a character level. Our model compares well to previously established normalization algorithms when evaluated on a diverse set of texts from Early New High German. We show that multi-task learning with additional normalization data can improve our model's performance further.

Introduction

Natural Language Processing (NLP) tools, such as part-of-speech taggers and parsers, are typically optimized for modern, standardized languages. When applied to historical documents, these tools often struggle due to the "abundance of variant spellings." In Early New High German, for instance, a single word like Frau (woman) might appear as fraw, vrowe, or frauwe, depending on the region, time period, or author's preference.

To address this, researchers employ spelling normalization as a preprocessing step to map archaic word forms to their modern equivalents. However, historical data is notoriously low-resource, leading to a traditional reliance on hand-crafted rules or simpler statistical models. This paper challenges the assumption that deep neural networks require massive datasets to be effective in this domain.

Methodology

The authors frame spelling normalization as a character-based sequence labeling task.

Alignment and Architecture

Because historical words and their modern counterparts often differ in length (e.g., vsfuret vs. ausfuhrt), an "iterated Levenshtein distance" algorithm is used to align characters. This process introduces a special "epsilon" label for deletions and merges characters for insertions.

The core architecture consists of a deep bi-directional Long Short-Term Memory (bi-LSTM) stack, three layers deep. Unlike standard unidirectional RNNs, bi-LSTMs process the sequence in both directions, capturing phonetic and orthographic context from both preceding and following characters.

Figure 1: The deep bi-LSTM architecture used for character-level sequence labeling.

Multi-task Learning (MTL)

To mitigate data scarcity, the model utilizes multi-task learning. It trains on a specific "target" text while simultaneously learning from auxiliary historical texts. Layers are shared across tasks, except for the final prediction layer, allowing the model to learn general spelling patterns while retaining the flexibility to handle specific dialectal variations.

Experimental Setup

The primary dataset is the Anselm Corpus, comprising 44 Early New High German texts (14th–16th century) representing various dialects. Additionally, the CELEX lexical database was used for experiments involving grapheme-to-phoneme mapping as an auxiliary task.

Results

The bi-LSTM model demonstrated superior performance, outperforming Norma (a popular rule-based tool) on 34 out of 44 texts. On average, the bi-LSTM achieved a 2.1 percentage point (pp) increase in word-level accuracy over Norma. Integrating multi-task learning (MTL) provided an additional 0.7 pp gain.

Figure 5: Detailed comparison of word-level accuracy across different texts and regions for Norma, CRF, and Bi-LSTM models.

In contrast, a Conditional Random Field (CRF) baseline using the same character-level setup performed poorly, trailing Norma by an average of 2.1 pp. This suggests that the LSTM's ability to learn long-term dependencies is critical to the success of the neural approach. Regarding data efficiency, while Norma remains more effective for very small datasets (under 1,000 training tokens) due to its use of external dictionaries, the bi-LSTM becomes the superior choice once this threshold is crossed.

Discussion

Several key insights emerged from this study:

  • Deep Learning on Small Data: Character-level embeddings keep the vocabulary size manageable, allowing bi-LSTMs to succeed even with datasets as small as ~7,000 tokens.
  • MTL as a Regularizer: Multi-task learning helps the model avoid overfitting to the idiosyncrasies of a single manuscript by exposing it to diverse spelling variations.
  • The Similarity Limit: The benefits of MTL are contingent on the relatedness of the auxiliary data; for example, using Dutch data to normalize German significantly reduced accuracy.

Conclusion

This work demonstrates that character-level bi-LSTMs are a powerful tool for historical spelling normalization, even in low-resource settings. While the current model relies on explicit character alignment, future work could explore encoder-decoder (seq2seq) architectures to eliminate this requirement. Additionally, identifying the most effective auxiliary tasks for MTL remains a promising area for further research.

Summary updated

Abstract

Natural-language processing of historical documents is complicated by the abundance of variant spellings and lack of annotated data. A common approach is to normalize the spelling of historical words to modern forms. We explore the suitability of a deep neural network architecture for this task, particularly a deep bi-LSTM network applied on a character level. Our model compares well to previously established normalization algorithms when evaluated on a diverse set of texts from Early New High German. We show that multi-task learning with additional normalization data can improve our model's performance further.
BibTeX
@inproceedings{bollmann2016improving,
  author = {Marcel Bollmann and Anders Søgaard},
  title = {{Improving historical spelling normalization with bi-directional LSTMs and multi-task learning}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=fjv6O2QBBj},
}
Figure