FORCE: A Benchmark Dataset for Foodborne Disease Outbreak and Recall Event Extraction from News

Sudeshna Jana, Manjira Sinha, Tirthankar DasguptaACL 2024

TL;DR

This paper introduces FORCE, a novel benchmark dataset and a multi-task learning model for automatically extracting critical information about foodborne disease outbreaks and recalls from news articles. By leveraging transformer-based document embeddings and a sequence labeling architecture, the proposed method aims to improve food safety monitoring and response.

What Problem Does This Paper Address?

Food safety incidents, including contamination and deliberate adulteration, are a growing concern, leading to widespread outbreaks and costly recalls. Understanding the origins and pathways of these incidents is crucial for prevention and mitigation. While existing efforts have focused on sentiment analysis or basic entity recognition, there's a significant gap in computational models that can perform deep linguistic analysis to identify and extract detailed information about food safety events, their causes, and affected parties from unstructured text like news reports. This lack of robust analytical tools is partly due to the scarcity of comprehensive, annotated datasets for training such models.

Methods

The authors propose a multi-tasking sequence labeling architecture built upon transformer-based document embeddings. Specifically, they fine-tune a BERT model to create "FoodSafety-BERT" (FS-BERT) for generating document embeddings. This is then integrated into a cascaded CNN-BiLSTM layer designed for two related tasks:

  1. Sentence Classification: Determining if a sentence describes a food recall, a disease outbreak, or is neutral.
  2. Sequence Labeling: Identifying and tagging specific phrases within the text that represent key entities and events, such as Target Organization (TO), Product Name (P), Infection Name (I), Safety Incident (SI), Cause of Incident (CI), and Number of People Affected (N).

The model is trained end-to-end using a joint loss function that linearly combines the cross-entropy losses for both tasks, with a mechanism to activate the sequence labeling loss only when a sentence is classified as relevant (recall or outbreak).

Datasets

The paper introduces FORCE, a new benchmark dataset for foodborne disease outbreak and recall event extraction. It comprises:

  • A corpus of 6000 regulatory articles from Food Safety News (FSN) covering outbreaks and recalls between 2011 and 2023.
  • A collection of approximately 2200 news articles from United States Food and Drug Administration (FDA) recall and outbreak announcements.

In total, the dataset consists of 8100 articles. The articles were manually annotated by multiple domain experts following a rigorous procedure to ensure inter-annotator agreement. The dataset will be publicly released.

Key Insights and Ideas

  • The Power of Multi-Task Learning: Combining sentence classification (identifying relevant sentences) with detailed sequence labeling (extracting specific event information) within a single model significantly improves performance. This suggests that these tasks are complementary and benefit from shared representations.
  • Transformer Embeddings for Domain Specificity: Fine-tuning BERT on a food safety corpus (creating FS-BERT) is crucial for capturing domain-specific nuances, leading to better performance than generic pre-trained models.
  • Data is King (and Scarce): The creation of the FORCE dataset is a major contribution, addressing the critical need for annotated data in this domain. The authors highlight that the scarcity of such data has been a bottleneck for developing advanced computational models.
  • Common Culprits: Analysis of the dataset reveals that Salmonella, E. coli, and Listeria monocytogenes are the most frequent causes of food recalls in the United States.

Key Results

  • The proposed multi-task BERT-CNN-BiLSTM model achieved an F1 score of 0.86 for sentence classification (food recall or disease outbreak), with a precision of 0.89.
  • For sequence labeling tasks, the multi-task model significantly outperformed several baseline single-objective models across most entity and event categories (TO, P, I, SI, CI, N). For instance, it achieved an F1 score of 0.88 for the 'N' (Number of People Affected) class.
  • While LLMs like LLAMA-2 7B and Mistral 7B were evaluated, they showed poorer performance, potentially due to a lack of domain-specific knowledge and difficulties in identifying precise phrase boundaries for incidents and causes.

Limitations and Future Work

  • LLM Limitations: The authors note that LLAMA-2 and Mistral 7B struggled with domain-specific concepts (e.g., specific pathogen names) and accurately identifying phrase boundaries for incidents and causes. This suggests that while LLMs are powerful, they may require more specialized fine-tuning or domain adaptation for highly specific tasks.
  • Future Work: The paper suggests that future work could focus on improving the identification of phrase boundaries for incidents and causes, and further exploring how to imbue models with deeper environmental domain knowledge. The authors also plan to apply their model to recent publications and conduct aggregate analysis of extracted insights.

Summary updated

BibTeX
@inproceedings{jana2024force,
  author = {Sudeshna Jana and Manjira Sinha and Tirthankar Dasgupta},
  title = {{FORCE: A Benchmark Dataset for Foodborne Disease Outbreak and Recall Event Extraction from News}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure