TL;DR
This paper tackles the challenge of normalizing historical texts, specifically 16th-century Middle French, by reducing spelling variations without modernizing the language. They introduce a novel approach using pre-trained language models and synthetic data generation, achieving significant improvements over existing methods and releasing a valuable new dataset for this low-resource task.
What Problem Does This Paper Address?
Digitizing and analyzing historical documents is crucial for fields like digital humanities and historical linguistics. However, a major hurdle is the inconsistent spelling and orthographic variations present in these texts, which often lack standardized conventions. While much research has focused on modernizing historical texts to make them compatible with modern computational tools, this approach loses valuable historical linguistic information. This paper addresses the need to normalize historical texts – reducing spelling variants to improve usability for computational analysis – without modernizing the language, thereby preserving historical wordforms for diachronic studies. The specific challenge is the scarcity of parallel data (historical text aligned with its normalized version) for Middle French, making supervised learning difficult.
Methods
The authors propose a multi-pronged approach leveraging pre-trained language models and synthetic data generation, inspired by low-resource machine translation techniques.
- Normalization as Machine Translation: They frame the normalization task as a sequence-to-sequence problem, similar to machine translation, where the input is the historical text and the output is its normalized counterpart.
- Synthetic Data Generation: To overcome the lack of parallel data, they employ two main strategies:
- Sequence-level Knowledge Distillation: They use a pre-trained model (teacher) to generate normalized versions of existing historical texts. This synthetic data is then used to train a new model (student) by interpolating the loss function with the original hand-crafted data. This helps prevent over-fitting to the small hand-crafted corpus.
- Transfer Learning and Back-translation: They fine-tune a generative causal language model (like Bloomz) on a small amount of target-side data to produce synthetic normalized text. This synthetic data is then used to train a normalization model via back-translation, where the model learns to translate from the synthetic normalized text back to the original historical text.
- Pre-trained Models: They utilize pre-trained encoder-decoder models (M2M100) and generative causal language models (Bloomz) as their foundation, fine-tuning them on their specific task.
Datasets
- Hand-crafted Parallel Corpus: A new, manually curated parallel corpus of 16th-century Middle French is introduced. This corpus consists of aligned segments of historical text and their normalized versions, created by experts following strict guidelines. This is the primary benchmark dataset.
- Manuscripts Corpus: This is the source side of the hand-crafted corpus, representing the original, non-normalized historical texts.
- Distillation Corpus: A synthetic parallel corpus generated using knowledge distillation.
- Generation Corpus: A synthetic parallel corpus generated using back-translation.
Key Insights and Ideas
- Normalization without Modernization is Crucial: The paper strongly advocates for preserving historical wordforms, highlighting that modernization loses valuable linguistic data for diachronic studies.
- Synthetic Data is a Game-Changer for Low-Resource Tasks: The success of their methods hinges on generating artificial parallel data, demonstrating its effectiveness in overcoming the scarcity of real-world annotated data for historical texts.
- Generative Models are Powerful for Data Augmentation: Fine-tuning large generative language models on even small amounts of target data allows them to produce high-quality synthetic data for downstream tasks.
- Knowledge Distillation and Back-translation Offer Complementary Benefits: Both synthetic data generation methods showed significant improvements, with back-translation (Generation) slightly outperforming distillation.
Key Results
- The "Generation" model, which uses synthetic data produced via back-translation, achieved the best performance, reaching an accuracy of 52.7% and a Word Error Rate (WER) of 5.1.
- Fine-tuning a pre-trained model (M2M100) on the hand-crafted corpus alone resulted in a significant increase in accuracy (36.1pts) compared to a simple copy baseline.
- The synthetic data approaches (Distillation and Generation) further improved performance over the hand-crafted only model.
- The proposed methods significantly outperformed a previously released model for early Modern French normalization (ModFr) in terms of WER.
Limitations and Future Work
- Limited Time Frame: Experiments were conducted on a specific variant of Middle French from 1536-1550, a relatively narrow period. Future work could explore broader historical periods.
- Model Selection: Only a limited number of pre-trained models were tested. The authors suggest that exploring more models, potentially those trained on related languages like Latin, could yield further improvements.
- Corpus Size: The hand-crafted corpus is relatively small, which is attributed to the high cost of expert annotation.
- Comparison of Synthetic Methods: The paper did not directly compare the Generation and Distillation approaches in terms of computational cost or data efficiency.
- Larger Models: The experiments used the smallest released versions of pre-trained models. Using larger models could potentially lead to better performance.
- Prompt Engineering: The authors believe that further exploration of prompt engineering techniques for generative models could unlock even greater potential.
- Modern French Data: Leveraging large amounts of publicly available modern French data and pre-trained LLMs is identified as a promising future direction.
Summary updated