Studying Language Variation Considering the Re-Usability of Modern Theories, Tools and Resources for Annotating Explicit and Implicit Events in Centuries Old Text

Stella Verkijk, Pia Sommerauer, Piek VossenNAACL 2024

TL;DR

This paper tackles the challenge of extracting events from centuries-old Dutch texts from the Dutch East India Company archives. The authors developed a new annotation scheme and ontology, demonstrating that modern NLP tools struggle with the significant linguistic variation in historical documents, highlighting the need for tailored solutions.

What Problem Does This Paper Address?

Event extraction, a core task in Natural Language Processing (NLP), aims to identify and classify events within text. While numerous datasets and systems exist for modern languages like English, applying these to historical texts presents a major hurdle. This paper focuses on the Dutch East India Company (VOC) archives, a vast collection of handwritten documents from the 17th and 18th centuries. This corpus is characterized by:

  • Extreme Linguistic Variation: No standardized spelling, diverse writing styles from numerous clerks, and intricate political/cultural conventions.
  • Historical Context: The language and the events described are often far removed from modern understanding, making interpretation difficult.
  • Scale: The corpus contains twenty-five million pages, making manual annotation and analysis a monumental task.

The core problem is the re-usability of modern NLP theories, tools, and resources for annotating and extracting events from such heterogeneous and historically distant text. The authors investigate how well existing NLP approaches, designed for modern Dutch and English, perform on this unique corpus and propose a tailored solution.

Methods

The paper outlines a multi-stage approach to build an event-centric Knowledge Graph (KG) from the VOC archives:

  1. Handwritten Text Recognition (HTR): Initial processing of scanned images to convert handwritten text into digital format. The authors used the Loghi system, acknowledging its challenges with historical documents, especially regarding spelling conventions and text region separation, leading to a significant Character Error Rate (CER).
  2. Manual Annotation: A crucial step involving expert historians annotating a subset of the digitized text.
  3. Annotation Scheme & Ontology: A new annotation task was designed, guided by a bespoke ontology developed by historians and linguists. This ontology models Dynamic Events (DEs) and Static Events (SEs) and their implications, aiming to capture the specific nuances of the VOC archives.
  4. Annotation Settings: To address the complexity, annotators worked in teams. They also employed a "check-task" where annotators reviewed each other's work, followed by adjudication with an expert linguist. This iterative process, including sharpening annotation guidelines, led to improved inter-annotator agreement (IAA).
  5. Ontological Resolutions: To further refine annotations and analyze IAA, two types of automatic resolutions were applied:
  6. Taxonomic Resolutions: Resolving disagreements on direct subclasses of the same event class by mapping them to a common superclass.
  7. Implicative Resolutions: Resolving disagreements between dynamic and static event labels by defaulting to the static event, as it represents a more conservative meaning.
  8. Automatic Baselines & Fine-tuning: After annotation, the authors fine-tuned Large Language Models (LLMs) for event trigger detection. They experimented with XLM-R and GysBERT-v2, a model specifically pre-trained on historical Dutch, including VOC data. They explored the "grokking" principle, training models for extended epochs beyond typical overfitting points.

Datasets

  • VOC Archives: The primary dataset, consisting of approximately seven million handwritten pages from the period 1610-1796. For this study, a subset of 62 pages from 6 different documents spanning 151 years (1626-1777) was annotated.
  • Existing NLP Datasets (for comparison/evaluation):
    • ACE (Automatic Content Extraction): Used as a benchmark for event extraction performance of modern models.
  • ERE (Event Representation Extraction): A lighter version of ACE.
  • FrameNet: A lexical resource for semantic role labeling.
  • PropBank & VerbNet: Resources focused on predicate-argument structures.
  • NewsReader pipeline: A system for extracting events from historical texts.

Key Insights and Ideas

  • Modern NLP Tools Struggle with Historical Variation: Existing models, even those trained on Dutch, perform poorly on early modern Dutch due to significant differences in lexicon, syntax, and discourse structure. This necessitates custom solutions.
  • The Importance of Domain-Specific Ontologies: A bespoke ontology, developed with domain experts (historians), is crucial for guiding annotation and capturing the specific types of events relevant to the VOC archives. This ontology helps manage the complexity and vastness of event types.
  • Iterative Annotation and Teamwork Enhance IAA: The annotation process, involving expert historians working in teams and iteratively refining guidelines, significantly improved inter-annotator agreement, highlighting the human-centric nature of this task.
  • Implicit Event References are Key: The authors emphasize the need to capture not only explicitly stated events but also implicitly evoked events, which are prevalent in the historical texts and crucial for a comprehensive understanding.
  • "Grokking" and Domain-Specific Pre-training Show Promise: While general LLMs struggle, models pre-trained on historical data (like GysBERT-v2) and trained beyond typical overfitting points ("grokking") show potential for better performance on this challenging task.

Key Results

  • Inter-Annotator Agreement (IAA): Achieved 84% for trigger detection, 86% for classification (of 80+ event types), and 72% for the combined task after annotation rounds and adjudication.
  • HTR Challenges: Character Error Rate (CER) for HTR was high (>10%), impacting downstream annotation.
  • Predicate Mapping: An experiment using existing resources for predicate mapping yielded more false positives (126) than true positives (95), indicating the inadequacy of off-the-shelf solutions.
  • Zero-shot POS-tagging: Models showed low accuracy, with GysBERT-v2 performing best but still below expert-level performance.
  • Fill-mask for Events: LLMs performed poorly, failing to predict relevant tokens for masked event triggers.
  • Automatic Baselines (Fine-tuning): GysBERT-v2 (pre-trained on historical Dutch) generally outperformed XLM-R, especially in precision, though XLM-R showed competitive recall. Training beyond overfitting ("grokking") improved performance.

Limitations and Future Work

  • HTR Quality: The quality of the HTR output remains a significant limitation, directly impacting the annotation process. Future work will focus on improving HTR.
  • Limited Annotated Data: The current annotated dataset is small, serving as a starting point. Future work includes creating more manually annotated data.
  • Model Comparison: A more thorough comparison of different LLMs, including GysBERT and GysBERT-v2, is planned.
  • Data Augmentation: Experimenting with automatic data augmentation techniques is a future direction.
  • Domain-Specific Lexicon: Developing a custom lexicon for pre-annotation is suggested as a more fruitful approach than using existing resources.
  • Grokking Principle: Further investigation into the grokking principle and its applicability to this task is warranted.

Summary updated

BibTeX
@inproceedings{verkijk2024studying,
  author = {Stella Verkijk and Pia Sommerauer and Piek Vossen},
  title = {{Studying Language Variation Considering the Re-Usability of Modern Theories, Tools and Resources for Annotating Explicit and Implicit Events in Centuries Old Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure