Closed-domain event extraction for hard news event monitoring: a systematic study

David Dukic, Filip Karlo Dosilovic, Domagoj Pluscec, Jan SnajderPeerJ Comput. Sci 2024 · OpenReview

TL;DR

This paper tackles the challenge of building robust news event monitoring systems by systematically evaluating state-of-the-art NLP models for "closed-domain main event extraction" (CDMEE) on hard news. They introduce a new, more realistic dataset (ASHNEE) and analyze the trade-offs between model accuracy and inference speed, offering practical guidance for deploying these systems.

What Problem Does This Paper Address?

The digital age bombards us with news, making it difficult to track truly important events – the "hard news" like conflicts or disasters. News event monitoring systems aim to automate this by extracting key information about events from text. While advanced NLP models exist, integrating them into real-world systems is tricky. Challenges include selecting the right models, ensuring they work reliably on messy, real-world data (not just clean datasets), and managing the computational demands for real-time processing. This paper addresses this gap by providing a systematic study on the practical application of CDMEE for hard news monitoring, focusing on these crucial integration issues.

Methods

The authors employ a two-stage pipeline approach for CDMEE, common in event extraction: Event Detection (ED) and Argument Extraction (AE). For ED, they compare a traditional linear SVM with several transformer-based pre-trained language models (PLMs) like RoBERTa and DeBERTa. For AE, they explore two main strategies: sequence labeling (treating it as a token classification task) and extractive question answering (QA), using PLMs fine-tuned for these tasks. They also investigate a binary classifier to distinguish hard news from other articles. A key aspect is their evaluation methodology, which includes using both clean and noisy datasets and analyzing the accuracy-speed trade-off on both CPU and GPU.

Datasets

  • DocEE: A large, existing dataset for document-level event extraction, used for training and initial evaluation. It contains hard and soft news event types but is noted as being "too clean" for realistic deployment.
  • ASHNEE (Automatically Scraped Hard News Event Extraction): A novel dataset created by the authors. It consists of articles scraped from numerous news outlets and includes ground-truth labels for document-level ED and AE on hard news, along with labels for soft news and event-less articles ("other"). This dataset is designed to provide a more realistic evaluation environment.

Key Insights and Ideas

  • Real-world data is messy: Off-the-shelf models trained on clean datasets struggle with the noise and variety of real-world news articles. The ASHNEE dataset highlights this by including non-hard news and event-less articles.
  • QA is king for Argument Extraction: For the AE task, extractive question answering (QA) models significantly outperform sequence labeling approaches, especially when dealing with the complexities of argument extraction in news.
  • Distilled models offer a sweet spot: Smaller, distilled versions of large PLMs (like DistilRoBERTa) can achieve performance comparable to their larger counterparts for Event Detection, while offering substantial gains in inference speed.
  • Pipeline robustness: The two-stage ED-AE pipeline is surprisingly robust to errors propagated from the ED stage to the AE stage, suggesting that high-performing ED models can make the overall system more reliable.
  • CPU vs. GPU is a major decision: The choice between CPU and GPU deployment has a dramatic impact on inference speed (orders of magnitude difference), with GPUs offering much higher throughput but at a higher cost.

Key Results

  • Event Detection (ED): RoBERTa (Large) achieved the highest F1 score (94.70%) on the clean DocEE dataset. However, on the noisy ASHNEE dataset, DeBERTa-v3 (Base) performed best with a macro F1 score of 93.20%. Distilled models like DistilRoBERTa showed competitive performance.
  • Argument Extraction (AE): DeBERTa-v3 (Base) achieved the highest F1 score (63.10%) for QA-based AE on the DocEE dataset. However, performance dropped significantly on the ASHNEE dataset (e.g., DeBERTa-v3 (Base) dropped by 17.70% F1), underscoring the need for fine-tuning on realistic data.
  • Accuracy-Speed Trade-off:
    • For ED, DistilRoBERTa (Base) on GPU can process 2,261 articles/second with a 94.20% F1 score, a slight drop from RoBERTa (Large) but a significant speedup.
  • For AE (QA-based), DeBERTa (Base) achieves 5 arguments/second with a 61.00% F1 score. Trading accuracy for speed, RoBERTa (Base) offers 46 arguments/second with a 56.00% F1 score.
  • Hard News Classification: A RoBERTa (Base) classifier achieved a macro F1 score of 96.82% for distinguishing hard news from soft news on the DocEE test set.

Limitations and Future Work

  • Language and Monolingual Setup: The study is limited to the English language. Future work could explore cross-lingual models to improve generalization and handle low-resource languages.
  • Inference Optimizations: The paper acknowledges the lack of advanced inference optimizations like task-specific distillation, weight quantization, or pruning, which could further boost speed.
  • Deployment Costs: The study focuses on technical aspects and doesn't deeply consider non-technical deployment costs (e.g., infrastructure, GPU adoption).
  • Future Directions: The authors suggest exploring multilingual models, further optimization techniques for inference, and more comprehensive cost analyses for deployment. They also highlight the need for research into reproducibility, replication, and deployment strategies for these models in production environments.

Summary updated

Abstract

News event monitoring systems allow real-time monitoring of a large number of events reported in the news, including the urgent and critical events comprising the so-called hard news. These systems heavily rely on natural language processing (NLP) to perform automatic event extraction at scale. While state-of-the-art event extraction models are readily available, integrating them into a news event monitoring system is not as straightforward as it seems due to practical issues related to model selection, robustness, and scale. To address this gap, we present a study on the practical use of event extraction models for news event monitoring. Our study focuses on the key task of closed-domain main event extraction (CDMEE), which aims to determine the type of the story’s main event and extract its arguments from the text. We evaluate a range of state-of-the-art NLP models for this task, including those based on pre-trained language models. Aiming at a more realistic evaluation than done in the literature, we introduce a new dataset manually labeled with event types and their arguments. Additionally, we assess the scalability of CDMEE models and analyze the trade-off between accuracy and inference speed. Our results give insights into the performance of state-of-the-art NLP models on the CDMEE task and provide recommendations for developing effective, robust, and scalable news event monitoring systems.
BibTeX
@inproceedings{dukic2024closed,
  author = {David Dukic and Filip Karlo Dosilovic and Domagoj Pluscec and Jan Snajder},
  title = {{Closed-domain event extraction for hard news event monitoring: a systematic study}},
  booktitle = {PeerJ Comput. Sci},
  year = {2024},
  url = {https://openreview.net/forum?id=60jzwZXCGt},
}
Figure