Improving historical spelling normalization with bi-directional LSTMs and multi-task learning

Marcel Bollmann, Anders Søgaard

Abstract

Historical documents present a significant challenge for modern Natural Language Processing (NLP) due to the lack of spelling standardization. This paper explores the use of deep bi-directional Long Short-Term Memory (bi-LSTM) networks for character-level spelling normalization. By framing the problem as a sequence labeling task and employing multi-task learning (MTL) to leverage data from related dialects, the proposed model outperforms traditional rule-based and CRF-based systems, even in data-scarce environments.

Introduction

Before the 18th century, spelling was largely non-standardized. A single word could appear in numerous variations depending on the scribe, region, or time period. For example, the German word Frau (woman) might be written as fraw, vrowe, or vorwe. To apply modern NLP tools like part-of-speech taggers or parsers to these texts, they must first be "normalized" into their modern equivalents.

The primary obstacle is the scarcity of annotated historical data. While deep learning typically requires massive datasets, historical corpora often contain only a few thousand tokens per text. This research investigates whether deep neural networks can effectively learn from such limited data.

Methodology: Character-Level Sequence Labeling

The authors treat spelling normalization as a character-based sequence labeling task. Instead of processing whole words, the model predicts the modern equivalent for each character in a historical word.

Architecture

The core of the system is a deep bi-LSTM with three hidden layers. This architecture processes character sequences in both forward and backward directions, allowing the model to capture the full context of surrounding letters when making a prediction.

The bi-LSTM architecture for character-level sequence labeling, featuring an embedding layer, stacked bi-LSTM layers, and a prediction layer.

Multi-Task Learning (MTL)

To address data sparsity, the researchers utilized Multi-Task Learning. The model is trained on a primary text while simultaneously learning from auxiliary texts (other historical documents). These tasks share internal layers, enabling the model to learn general patterns of historical spelling variation that transcend specific dialects. This approach acts as a regularizer, preventing the model from overfitting to the idiosyncrasies of a single scribe.

Experimental Setup

The models were evaluated using the Anselm corpus, which contains 44 semi-parallel German manuscripts from the 14th to 16th centuries. These texts represent various dialects, including East Central and West Upper German. Each dataset is relatively small, ranging from approximately 4,200 to 13,200 tokens. For evaluation, the first 1,000 tokens of each text were used as a test set, with the remainder serving as training data.

Results and Discussion

The bi-LSTM approach demonstrated clear superiority over established baselines. On average, the basic bi-LSTM achieved a word-level accuracy of 79.90%, compared to 77.83% for the rule-based tool Norma and 75.73% for a Conditional Random Field (CRF) tagger.

The Data Threshold

A key finding was the relationship between training set size and model performance. As shown in the learning curves, simple rule-based systems like Norma perform better when training data is extremely limited (fewer than 500 tokens). However, once the training set exceeds approximately 1,000 tokens, the bi-LSTM consistently takes the lead.

Learning curves showing word-level accuracy across different training set sizes for Norma, CRF, and Bi-LSTM models.

Impact of Multi-Task Learning

Integrating MTL further enhanced performance, raising the average accuracy to 80.55%. In some instances, MTL provided an improvement of up to 9.6 percentage points over the baseline. However, the effectiveness of MTL depends on the linguistic relatedness of the auxiliary data; using unrelated data (e.g., Dutch data for German normalization) was found to be detrimental.

Conclusion

This study demonstrates that deep bi-LSTMs are highly effective for historical spelling normalization, provided a modest amount of training data (~1,000 tokens) is available. The character-level approach keeps the vocabulary manageable, while multi-task learning allows the model to generalize across dialects. Future work may explore encoder-decoder architectures to eliminate the need for explicit character alignment and investigate broader auxiliary tasks to further improve linguistic robustness.

Summary updated

BibTeX
@inproceedings{bollmann2016improving,
  author = {Marcel Bollmann and Anders Søgaard},
  title = {{Improving historical spelling normalization with bi-directional LSTMs and multi-task learning}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=fjv6O2QBBj},
}
Figure