Abstract: Training neural sequence-to-sequence models with simple token-level log-likelihood is now a standard approach to historical text normalization, albeit often outperformed by phrase-based models. Policy gradient training enables direct optimization for exact matches, and while the small datasets in historical text normalization are prohibitive of from-scratch reinforcement learning, we show that policy gradient fine-tuning leads to significant improvements across the board. Policy gradient training, in particular, leads to more accurate normalizations for long or unseen words.
Introduction
Historical text normalization—the task of mapping archaic spellings (e.g., "vpon") to modern forms ("upon")—is a critical preprocessing step for analyzing historical documents. While neural sequence-to-sequence (Seq2Seq) models are commonly used, they are typically trained using Maximum Likelihood Estimation (MLE). MLE operates at the character level, attempting to predict the next character correctly. This creates a discrepancy: in evaluation, a word with one incorrect character is often just as wrong as a word with five. MLE lacks the holistic word-level perspective required for optimal normalization.
Methodology: Policy Gradient Fine-tuning
To address this, the authors propose a two-stage training pipeline. First, an LSTM-based encoder-decoder model is pre-trained using standard MLE for 10 epochs. This provides a stable baseline, which is necessary as reinforcement learning (RL) is notoriously difficult to train from scratch on the small datasets typical of historical linguistics.
Second, the model is fine-tuned using the REINFORCE algorithm. Instead of minimizing character-level log-likelihood, the model treats the generated word as a "policy" and receives a "reward" based on how close that word is to the ground truth.
The primary reward function used is the negative Levenshtein distance. As shown in the training curves, Levenshtein distance consistently outperformed other metrics like Hamming or Jaro-Winkler during the fine-tuning process. To stabilize training, the authors employed variance reduction by using the mean reward of sampled hypotheses as a baseline and applied smoothing to prevent premature overconfidence.
Experimental Setup
The approach was evaluated across six diverse historical datasets: English (1386–1698), German (14th–16th c.), Hungarian (1440–1541), Icelandic (15th c.), Slovene (1750–1840s), and Swedish (1527–1812). These datasets vary significantly in size and "Invariance Across Time" (IAT)—the percentage of words that haven't changed spelling at all.
Key Results
The "MLE + Policy Gradient" (PG) approach yielded significant improvements over the MLE baseline for every language tested.
As seen in the results table, Slovene saw a 31% error reduction, while English saw a 20% reduction. The improvements were not just quantitative; the RL-tuned models were qualitatively better at handling long words and avoiding "over-normalization" of words that should remain identical to their historical forms.
Discussion and Insights
The success of policy gradient fine-tuning highlights several key advantages:
- Delayed Rewards Matter: By using rewards that trigger only after a full word is generated, the model learns to optimize for the final outcome rather than just the immediate next character.
- Mitigating Error Propagation: Standard Seq2Seq models often suffer from cascading errors. The RL approach evaluates the final string holistically, making it significantly more robust for long words.
- Generalization to Unseen Words: A striking finding was that RL fine-tuning performed much better on words not present in the training set, suggesting the model learns more generalizable transduction patterns.
Conclusion
While this study utilized LSTM architectures, the benefits of policy gradient fine-tuning are likely orthogonal to the underlying model and could be applied to Transformers. Despite the constraints of small historical datasets, RL fine-tuning proves to be a powerful tool for bridging the gap between training objectives and evaluation metrics.
Summary updated