A Neural Model for Part-of-Speech Tagging in Historical Texts

Christian HardmeierCOLING 2016

TL;DR

This paper introduces a character-level neural network for Part-of-Speech (POS) tagging of historical texts, bypassing the need for explicit spelling normalization. The model achieves performance comparable to traditional two-stage approaches, demonstrating its potential for handling linguistic variations in historical documents.

What Problem Does This Paper Address?

Processing historical texts with modern NLP tools is challenging due to significant linguistic divergence, including inconsistent orthography, morphology, and syntax. Traditional approaches often rely on manual annotation of historical data or a two-step process: first, normalizing historical spellings to modern ones, and then applying standard NLP tools. This paper tackles the problem of directly tagging historical texts without a separate spelling normalization stage, aiming to simplify the pipeline and potentially improve accuracy by handling original spellings.

Methods

The core of the proposed solution is a character-level recurrent neural network (RNN) architecture, inspired by Ling et al. (2015). The model processes sentences character by character, generating dense vector representations that capture character and contextual information. Crucially, at word boundaries, these character-level representations are combined to predict POS tags for entire words.

A key innovation is the training objective, which combines two error terms:

  • POS tagging error (EPOS): Standard cross-entropy loss on a modern, POS-tagged corpus.
  • Normalization error (Enorm): Squared error between vector representations of historical text and its normalized form, ensuring that the model learns similar representations for original and normalized spellings.

This dual objective allows the model to learn POS tagging while simultaneously learning to handle historical variations. The model is trained using minibatch stochastic gradient descent with gradient clipping.

Datasets

The experiments utilize the following datasets:

  • Modern Corpora:
    • Swedish: Stockholm-Umeå Corpus (SUC) v2.0, a balanced collection of Swedish texts from the 1990s.
  • German: NEGRA corpus, a standard POS-tagged corpus for German.
  • Historical Corpora:
    • Swedish: Gender and Work corpus, a collection of early modern Swedish texts (1550-1800) used for verb identification.
  • German: GerManC corpus (gold-standard portion), a corpus of early modern German (1650-1800) with normalized spelling, lemmas, and POS tags.

The paper also mentions using a normalized version of the historical corpora for training the normalization error component.

Key Insights and Ideas

  • Direct Historical Tagging: The most significant insight is that a character-level neural network can effectively perform POS tagging on historical texts without explicit spelling normalization, simplifying the NLP pipeline.
  • Dual Training Objective: Combining POS tagging accuracy with a representation similarity constraint (Enorm) is crucial for enabling the model to learn from historical data while maintaining performance. This suggests that learning to "understand" historical variations is as important as the tagging task itself.
  • Flexibility of Neural Models: The neural approach offers greater flexibility than pipeline methods, potentially allowing for more nuanced handling of diachronic linguistic changes.
  • Cross-Lingual Potential: The authors suggest that the versatility of vector-space embeddings could be leveraged by incorporating data from closely related contemporary languages (e.g., Icelandic for Swedish) to improve historical language processing.

Key Results

  • Swedish Verb Identification: The proposed neural tagger achieved an F-score of 0.8529 on the test set, which is competitive with, and slightly better than, a strong baseline involving SMT-based spelling normalization (0.855 reported by Pettersson).
  • German POS Tagging: For German, the neural system achieved an F-score of 0.8782 on the test set, performing almost on par with the SMT normalization baseline.
  • Robustness to Historical Spellings: The adapted Swedish neural systems showed slightly better performance on historical spellings compared to a standard HunPos tagger without normalization.
  • Qualitative Observations:
    • For Swedish, both the neural tagger and the baseline struggled with similar issues, particularly confusing common nouns (NN) with verbs (VB), and verbs with participles (PC).
  • The neural tagger was more prone to assigning the "foreign word" (UO) tag, sometimes incorrectly, potentially due to the influence of modern English in the training data.
  • For German, the neural tagger handled certain historical forms (e.g., possessive pronouns) better than the HunPos baseline. However, both systems struggled with homonymous verb forms (infinitives vs. first-person plural).

Limitations and Future Work

  • Performance Gap: While competitive, the neural tagger's performance was still slightly lower than the best SMT normalization baselines in some cases.
  • Swedish Specifics: The authors note that for Swedish, addressing deeper linguistic issues beyond orthography, such as morphology and syntax, will be necessary for significant improvements.
  • German Gold Standard: The inconsistency of the German gold standard for certain verb forms makes it difficult to fully evaluate performance in those specific cases.
  • Future Work:
    • Exploiting data from closely related languages to improve historical language models.
  • Further investigation into the regularization effect of the normalization error term.
  • Developing more targeted approaches to handle specific diachronic changes in morphology and syntax.

Summary updated

BibTeX
@inproceedings{hardmeier2016neural,
  author = {Christian Hardmeier},
  title = {{A Neural Model for Part-of-Speech Tagging in Historical Texts}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure