Abstract
Historical text normalization is a critical preprocessing step for applying modern NLP tools to archival documents. However, the scarcity of annotated historical data often limits model performance. While Multi-task Learning (MTL) is frequently proposed as a solution for low-resource settings, its generalizability across diverse languages and data scales remains poorly understood. This study evaluates MTL across 10 datasets spanning 8 languages and various historical periods. Our findings reveal a distinct "threshold effect": MTL provides universal benefits when training data is extremely scarce (approx. 5,000 tokens) but often becomes detrimental as the target dataset size increases.
Introduction
Historical documents provide invaluable insights into linguistic and cultural evolution, yet their non-standardized orthography poses a significant barrier to automated analysis. Historical text normalization aims to map archaic spellings (e.g., feüchter) to their modern equivalents (feuchter). Because creating gold-standard datasets requires specialized expertise, most available corpora are small and highly variable.
Methodology
The researchers utilized a state-of-the-art character-level encoder-decoder model with an attention mechanism. In the multi-task setup, two datasets share a bidirectional LSTM encoder and a unidirectional LSTM decoder, but maintain separate final prediction layers. This architecture encourages the model to learn shared character transformation patterns across different languages or tasks.
Experimental Setup and Datasets
The study covers a broad linguistic range, including German, English, Hungarian, Icelandic, Spanish, Portuguese, Slovene, and Swedish. The datasets vary significantly in size, ranging from roughly 24,000 to over 230,000 training tokens.
The experiments were conducted under two primary conditions:
- Full Data: Utilizing the maximum available training tokens for each language.
- Sparse Data: Artificially limiting the target training set to 5,000 tokens to simulate extreme low-resource conditions.
Results: The "Threshold" Effect
The primary finding of this research is that the success of MTL is predominantly determined by the size of the target dataset rather than linguistic similarity.
In the Sparse Scenario, MTL improved accuracy for every dataset tested. For instance, English normalization accuracy rose from 75.43% to 81.02%. Conversely, in the Full Scenario, MTL often introduced noise, leading to performance drops (e.g., English accuracy falling from 93.95% to 93.46%).
Interestingly, for English, using modern "noisy" text from Twitter as an auxiliary task proved even more effective (81.72% accuracy) than using other historical languages. This suggests that the model benefits from learning general patterns of spelling variation, regardless of whether the source is a 15th-century manuscript or a modern social media post.
Conclusion
Contrary to common NLP assumptions, MTL is not a universal performance booster for text normalization. It serves as a vital "rescue" technique for extremely low-resource settings but loses its efficacy as more target-specific data becomes available. Future research should explore many-to-one MTL configurations and the integration of syntactic context to resolve ambiguities that word-by-word models cannot currently address.
Summary updated