TL;DR
This paper evaluates three methods for normalizing spelling in historical texts across five languages. They find that a character-based statistical machine translation (SMT) approach generally performs best, significantly improving the accuracy of text analysis tools for historical documents. This work is crucial for making historical texts more accessible to modern NLP techniques.
What Problem Does This Paper Address?
Historical texts present a significant challenge for modern Natural Language Processing (NLP) tools. This is due to several factors:
- Data Sparsity: Historical language data is often scarce and not annotated, which is essential for training NLP models.
- Language Evolution: Languages change over time, meaning NLP models trained on modern text may perform poorly on older documents.
- Spelling Variation: Before standardized spelling conventions, historical texts exhibit substantial variation in spelling, even within the same document or by the same author.
To overcome these hurdles, researchers often employ spelling normalization – converting historical spellings to a modern, standardized form. This allows existing NLP tools designed for modern languages to be applied to historical texts. However, evaluating different normalization methods across multiple languages and comparing their effectiveness has been limited. This paper aims to fill that gap by presenting a multilingual evaluation of three distinct spelling normalization approaches.
Methods
The authors evaluate three spelling normalization methods:
- Filtering Approach: This method relies on a parallel training corpus where historical word forms are mapped to their modern spellings. When a token is encountered that exists in the training data, it's replaced with its most frequent modern spelling. Tokens not found in the training data are left unchanged.
- Levenshtein-based Approach: This technique uses edit distance (specifically, Levenshtein distance) to find the closest modern spelling for a historical word form from a modern dictionary or corpus. It can be enhanced by using parallel training data to learn weights for specific edit operations, making it more context-aware. A key advantage is that it doesn't strictly require parallel historical/modern data for training, only a modern dictionary.
- Character-based Statistical Machine Translation (SMT) Approach: This method treats spelling normalization as a translation problem. Instead of translating words, it translates character sequences. This is particularly effective for handling spelling variations without needing large amounts of parallel data, as character sets are much smaller than word sets. The system uses the Moses SMT toolkit and can employ different alignment techniques (GIZA++ or m2m-aligner) and character n-gram models.
Datasets
The study utilizes historical and modern language resources across five languages: English, German, Hungarian, Icelandic, and Swedish. For each language, they use:
- Training, Tuning, and Evaluation Corpora: These consist of historical texts with manually normalized spellings. The specific corpora vary by language (e.g., ICAMET for English, GerManC for German, HGDS for Hungarian, IcePaHC for Icelandic, GaW for Swedish).
- Modern Language Resources: These are used for dictionary lookups (Levenshtein) and language models (SMT). Examples include the British National Corpus (BNC) for English, Parole for German, Szeged Treebank for Hungarian, and SUC for Swedish.
The paper notes that the availability and size of these parallel historical/modern datasets vary significantly across languages, which is a common challenge in historical language technology research.
Key Insights and Ideas
- SMT's Multilingual Strength: The character-based SMT approach consistently shows strong performance across most languages, highlighting its adaptability.
- Levenshtein's Data Efficiency: The Levenshtein-based approach is valuable when parallel historical/modern data is scarce, as it can function with just a modern dictionary.
- Filtering's Simplicity and Effectiveness: The filtering approach, while simple, is surprisingly effective, especially for high-frequency words, and significantly improves accuracy over the baseline.
- No One-Size-Fits-All: No single method is universally best for all languages. The optimal choice depends on the language, data availability, and specific characteristics of the historical text.
- Context Matters (Subtly): While SMT is character-based, the study found that simple unigram models often performed as well as or better than bigram models, suggesting that complex character-level context might not always be necessary for spelling normalization.
- Historical vs. Modern Spelling Differences: The study highlights that even seemingly similar languages (like Icelandic) can have subtle but persistent spelling differences that require careful normalization.
Key Results
The paper presents results in terms of normalization accuracy (percentage of tokens matching the modern gold standard) and character error rate (CER).
- Baseline Improvement: All three methods significantly improve normalization accuracy compared to the baseline (which represents tokens already in modern spelling).
- SMT Dominance: For four out of five languages (English, German, Icelandic, Swedish), the SMT-based approach (specifically with GIZA unigram models) achieved the highest accuracy. For example, in English, SMT reached 94.3% accuracy with a CER of 0.07, compared to a baseline of 75.8% accuracy and 0.26 CER.
- Levenshtein + Filter Synergy: The combination of the filtering approach and the Levenshtein-based approach (Lev+filter) yielded the highest accuracy for Icelandic (90.8% accuracy, 0.10 CER), outperforming other methods for that language.
- Hungarian Anomaly: Hungarian showed the lowest baseline accuracy (17.1%) and the highest CER (0.85), indicating a greater divergence between its historical and modern spellings. While SMT improved this significantly, it still lagged behind its performance in other languages.
- Precision/Recall/F-score: Table 7 shows that SMT generally achieves high precision and recall, leading to the best F-scores for most languages. The Lev+filter approach also performs very well, especially for Icelandic.
Limitations and Future Work
- Language-Specific Performance: The authors acknowledge that the varying performance across languages is not fully explained and would benefit from further investigation into why certain approaches work better for specific languages. This is complicated by differences in corpus size, genre, and time periods.
- Extrinsic Evaluation: The current evaluation is intrinsic (measuring normalization accuracy). For historians, an extrinsic evaluation (measuring the impact of normalization on downstream tasks like tagging and parsing) would be more valuable.
- Word-Level vs. Context: The current methods treat words in isolation. Future work could explore handling grammatical and structural differences between historical and modern language as part of the normalization process.
- Data Availability: The study highlights the ongoing challenge of data sparseness for historical languages, which limits the ability to create perfectly comparable datasets across languages.
Summary updated