Introduction
Lemmatization—the task of mapping word forms (e.g., "running") to their canonical base forms (e.g., "run")—is a fundamental step in natural language processing. While considered largely solved for modern standard languages like English, it remains a significant challenge for historical and non-standard languages. Researchers have recently developed a state-of-the-art neural lemmatization system that excels in these difficult contexts. By jointly training a character-level encoder-decoder model with a word-level language modeling objective, the system effectively navigates extreme spelling variation and morphological complexity without requiring expensive manual annotations such as Part-of-Speech (POS) tags.
The Challenge of Non-Standard Languages
Historical texts and low-resource dialects present three primary hurdles for traditional lemmatizers:
- Morphological Complexity: Many languages feature intricate inflectional patterns that are difficult to capture with simple rules.
- Token-Lemma Ambiguity: Homographs (words with identical spellings but different base forms) require contextual understanding to resolve. For example, "living" could be a noun or a participle of "live."
- Spelling Variation: Before orthographic standardization, a single lemma could appear in dozens of variants. In Middle Dutch, the lemma jaar ("year") might be spelled as iare, ior, or jaer.
Traditional "edit-tree" methods, which treat lemmatization as a classification task of transformation rules, often fail when data is sparse or inconsistent. Modern neural models typically rely on POS tags to resolve ambiguity, but such tags are rarely available for niche historical corpora.
Hierarchical Joint Learning Architecture
The proposed solution utilizes a Hierarchical Sentence Encoder integrated into a character-level Encoder-Decoder architecture.
The model processes tokens character-by-character to remain robust against spelling variations. These character representations are aggregated into word-level vectors, which then pass through a bidirectional RNN to capture sentence-level context.
The critical innovation is the Joint Learning objective. In addition to generating the correct lemma, the model is simultaneously tasked with a Language Modeling (LM) loss—predicting the preceding and succeeding words in the sentence. This auxiliary task forces the sentence encoder to develop "meaning-aware" representations, providing the benefits of context-awareness (traditionally provided by POS tags) using only raw text.
Experimental Setup
The model was evaluated across two diverse categories:
- Historical Languages: A new dataset of 8 corpora, including Middle Dutch (administrative, literary, and religious), Middle Low German, Medieval French, Medieval Latin, and Historical Slovene.
- Standard Languages: 20 typologically diverse languages from the Universal Dependencies (v2.2) corpus to ensure generalizability.
Performance Analysis
The joint-learning model, referred to as Sent-LM, demonstrated substantial improvements over baseline methods.
On historical datasets, Sent-LM achieved an average accuracy of 94.0%, representing a 7.9% relative improvement over standard neural context models and a 30.72% improvement over traditional edit-tree approaches. The impact was most pronounced on ambiguous tokens, where the LM loss improved accuracy by over 15% relative to the baseline. Furthermore, the system outperformed previous specialized models on the Gysseling Middle Dutch corpus, particularly in handling unknown and ambiguous tokens.
Insights and Future Directions
Probing experiments revealed that the joint-LM model's internal representations are significantly better at predicting POS tags and morphological categories than standard models, even without explicit training on those features. This suggests that the LM loss acts as a "free" supervisor, allowing the model to implicitly learn grammatical structures.
However, a limitation was observed regarding "unknown" tokens (words not seen during training), where the word-level LM objective occasionally hindered performance by predicting generic "unknown" tags. Future work may involve moving toward character-level language modeling for the joint objective. This shift could help the model generalize its contextual knowledge to entirely new word forms, further bridging the gap for low-resource historical dialects.
Summary updated