Multi-task learning for historical text normalization: Size matters

Marcel Bollmann, Anders Søgaard, Joachim BingelDeepLo @ ACL 2018 · OpenReview

Abstract

Historical text normalization is a critical preprocessing step for applying modern NLP tools to archival documents. However, the scarcity of annotated historical data often limits model performance. While Multi-task Learning (MTL) is frequently proposed as a solution for low-resource settings, its generalizability across diverse languages and data scales remains poorly understood. This study evaluates MTL across 10 datasets spanning 8 languages and various historical periods. Our findings reveal a distinct "threshold effect": MTL provides universal benefits when training data is extremely scarce (approx. 5,000 tokens) but often becomes detrimental as the target dataset size increases.

Introduction

Historical documents provide invaluable insights into linguistic and cultural evolution, yet their non-standardized orthography poses a significant barrier to automated analysis. Historical text normalization aims to map archaic spellings (e.g., feüchter) to their modern equivalents (feuchter). Because creating gold-standard datasets requires specialized expertise, most available corpora are small and highly variable.

Methodology

The researchers utilized a state-of-the-art character-level encoder-decoder model with an attention mechanism. In the multi-task setup, two datasets share a bidirectional LSTM encoder and a unidirectional LSTM decoder, but maintain separate final prediction layers. This architecture encourages the model to learn shared character transformation patterns across different languages or tasks.

Experimental Setup and Datasets

The study covers a broad linguistic range, including German, English, Hungarian, Icelandic, Spanish, Portuguese, Slovene, and Swedish. The datasets vary significantly in size, ranging from roughly 24,000 to over 230,000 training tokens.

Overview of the 10 historical datasets used in the study, showing the diversity in time periods and token counts.

The experiments were conducted under two primary conditions:

  1. Full Data: Utilizing the maximum available training tokens for each language.
  2. Sparse Data: Artificially limiting the target training set to 5,000 tokens to simulate extreme low-resource conditions.

Results: The "Threshold" Effect

The primary finding of this research is that the success of MTL is predominantly determined by the size of the target dataset rather than linguistic similarity.

Comparison of MTL performance gains/losses. The left heatmap (sparse data) shows widespread improvements (red), while the right (full data) shows performance degradation (blue) as data becomes more abundant.

In the Sparse Scenario, MTL improved accuracy for every dataset tested. For instance, English normalization accuracy rose from 75.43% to 81.02%. Conversely, in the Full Scenario, MTL often introduced noise, leading to performance drops (e.g., English accuracy falling from 93.95% to 93.46%).

Interestingly, for English, using modern "noisy" text from Twitter as an auxiliary task proved even more effective (81.72% accuracy) than using other historical languages. This suggests that the model benefits from learning general patterns of spelling variation, regardless of whether the source is a 15th-century manuscript or a modern social media post.

Conclusion

Contrary to common NLP assumptions, MTL is not a universal performance booster for text normalization. It serves as a vital "rescue" technique for extremely low-resource settings but loses its efficacy as more target-specific data becomes available. Future research should explore many-to-one MTL configurations and the integration of syntactic context to resolve ambiguities that word-by-word models cannot currently address.

Summary updated

Abstract

Historical text normalization suffers from small datasets that exhibit high variance, and previous work has shown that multi-task learning can be used to leverage data from related problems in order to obtain more robust models. Previous work has been limited to datasets from a specific language and a specific historical period, and it is not clear whether results generalize. It therefore remains an open problem, when historical text normalization benefits from multi-task learning. We explore the benefits of multi-task learning across 10 different datasets, representing different languages and periods. Our main finding—contrary to what has been observed for other NLP tasks—is that multi-task learning mainly works when target task data is very scarce.
BibTeX
@inproceedings{bollmann2018multi,
  author = {Marcel Bollmann and Anders Søgaard and Joachim Bingel},
  title = {{Multi-task learning for historical text normalization: Size matters}},
  booktitle = {DeepLo @ ACL},
  year = {2018},
  url = {https://openreview.net/forum?id=pigw3xCdzp9},
}
Figure