TL;DR
This paper introduces BRAD, a novel dataset for event extraction specifically designed for historical texts, focusing on Black rebellions in 19th-century African American newspapers. They demonstrate that current state-of-the-art event extraction models perform significantly worse on historical texts compared to modern ones, highlighting the need for specialized approaches and datasets.
What Problem Does This Paper Address?
Understanding historical events is crucial for comprehending contemporary society and politics. The field of Natural Language Processing (NLP) has made strides in "event extraction" (EE), a task that identifies key words (triggers) and their associated arguments (participants, locations, times) that describe an event. However, existing EE datasets and models are largely trained on modern text, which differs significantly from historical documents. Historical texts present unique challenges: word meanings evolve over time, religious discourse is prevalent, and the specific topics of interest to humanities scholars (like Black rebellions) are not well-represented in current benchmarks. This gap hinders the application of NLP to historical research, preventing scholars from leveraging the vast amount of digitized historical documents for nuanced analysis.
Methods
The authors tackle this problem by:
- Creating a New Dataset (BRAD): They curated a dataset of 151 documents from the African American newspaper corpus, specifically focusing on 19th-century Black rebellions. This involved:
- Expert Curation: A humanities expert identified relevant documents using keyword matching and manual review.
- Annotation Schema: They defined 5 entity types (Person, Organization, Geographical-Political Entities, Time, Document) and 12 event types (e.g., Humanity_Deprive, Law_Approve, Conflict_Attack, Justice_Execute), along with 6 argument roles.
- Annotation Process: Two graduate students annotated the data, achieving high inter-annotator agreement (Cohen's Kappa scores of 0.81 and 0.82 for entity mentions and event triggers, respectively). A humanities expert resolved conflicts.
- Evaluating Existing Models: They evaluated two state-of-the-art EE models, DyGIE++ and OneIE, which leverage pre-trained BERT models.
- Investigating Model Adaptation: They explored fine-tuning the BERT model on the African American newspaper corpus (excluding the BRAD documents) to better adapt it to the historical language.
Datasets
- BRAD (Black Rebellions in African Diaspora): A newly introduced dataset specifically for event extraction from historical texts. It contains 151 documents, 5 entity types, 12 event types, and 6 argument roles, focusing on 19th-century Black rebellions.
- ACE 2005: A standard, widely used dataset for event extraction, used here for comparison to highlight the performance gap between modern and historical texts.
Key Insights and Ideas
- Historical Text is Fundamentally Different: The performance drop of current EE models on BRAD (compared to ACE 2005) is substantial, underscoring that historical language, with its evolving semantics and unique discourse patterns, requires specialized handling.
- Fine-tuning BERT Helps, But Isn't Enough: Fine-tuning BERT on a relevant historical corpus (African American newspapers) significantly improves performance on BRAD, demonstrating the benefit of domain-specific pre-training. However, performance still lags considerably behind that on modern datasets.
- Language Evolution is a Major Hurdle: The authors highlight how word meanings change and religious language is more prevalent in historical texts, posing challenges for models trained on contemporary language.
- Annotation Ambiguity in Historical Context: The paper points out that interpreting historical texts can be inherently ambiguous, even for human experts, due to context-dependent word meanings and the need for specific historical knowledge.
Key Results
- Performance Gap: State-of-the-art EE models (DyGIE++ and OneIE) achieved significantly lower F1 scores on BRAD compared to ACE 2005 across all sub-tasks (Entity Mention Detection, Event Trigger Identification/Classification, Event Argument Identification/Classification). For example, the F1 score for Event Trigger Classification dropped from around 77% on ACE 2005 to around 50-60% on BRAD.
- Fine-tuning Benefits: Fine-tuning BERT on the African American newspaper corpus improved the performance of both models on BRAD. For instance, OneIE's F1 score for Event Trigger Classification increased from 59.4% to 63.9% after fine-tuning.
- Human Performance: Human annotators achieved F1 scores of 95.43% for Entity, 88.3% for Trig-C, and 79.8% for Arg-C on BRAD, indicating a significant gap between current models and human capabilities.
Limitations and Future Work
- Dataset Size: While BRAD is a valuable contribution, the authors acknowledge that it is relatively small (151 documents).
- Model Adaptation: Current models still perform poorly on historical texts, indicating a need for more robust adaptation strategies beyond simple fine-tuning.
- Future Work: The authors plan to enlarge the BRAD dataset with more annotated documents and event types. They also suggest further research into developing more effective EE models for historical texts, potentially by incorporating more sophisticated contextual understanding or specialized linguistic features.
Summary updated