Explicit, Implicit, and Scattered: Revisiting Event Extraction to Capture Complex Arguments

Omar Sharif, Joseph Gatto, Madhusudan Basak, Sarah Masud PreumEMNLP 2024 · DOI

Introduction

Traditional Event Extraction (EE) methodologies have long relied on a "span-extraction" paradigm, where models identify events and highlight contiguous strings of text to fill argument roles. While effective for structured news reporting, this approach falters in the nuanced, informal landscape of social media. Researchers from Dartmouth College address this gap by introducing DiscourseEE, a framework that treats argument extraction as a generative task, enabling the capture of information that is not explicitly stated or is distributed across a document.

Redefining Argument Extraction

The core challenge in informal discourse, such as online health forums, is that critical details are often non-explicit. The authors categorize arguments into three distinct types:

  1. Explicit: Contiguous text spans (e.g., "prostate cancer").
  2. Implicit: Information that must be inferred from context (e.g., a goal dosage of "0mg" for a patient attempting to quit a medication).
  3. Scattered: Details that must be synthesized from multiple parts of a post or an entire comment thread.
Figure 1: Conceptual overview of explicit, implicit, and scattered argument types in medical discourse.

The DiscourseEE Dataset

To benchmark this new approach, the researchers developed the DiscourseEE dataset, sourced from Reddit communities focused on Opioid Use Disorder (OUD). The dataset contains 7,464 argument annotations across three event types: Taking MOUD, Return to Usage, and Tapering.

Significantly, the dataset reveals that the vast majority of information in these peer-to-peer interactions is non-explicit: 51.2% of arguments are implicit and 17.4% are scattered. This complexity renders traditional extractive models, which achieved only a 17.13 RM_F1 score, largely obsolete for this domain.

Figure 6: A complete example of event extraction from a post-comment pair in the DiscourseEE dataset, showing core and specific arguments.

Generative Modeling and Evaluation

The proposed framework shifts from "finding the span" to "generating the answer." By utilizing Large Language Models (LLMs) and instruction-tuned models like FLAN-T5, the system can describe arguments in natural language.

Because generative models may use synonyms (e.g., "runny nose" vs. "nasal discharge"), the authors implemented Relaxed Matching (RM_F1). This metric uses BERT embeddings to calculate semantic similarity, considering a response correct if it meets a 0.75 similarity threshold with the ground truth. This shift more accurately reflects model performance compared to rigid "Exact Match" standards.

Key Findings

  • The Hidden Majority: Nearly 70% of vital health information is missed by traditional extractive models because it is not written verbatim.
  • Model Performance: GPT-4 emerged as the top performer with a 41.98 RM_F1 score. However, even advanced LLMs struggle with implicit reasoning, achieving only 36.53% recall for implicit arguments compared to 55.14% for explicit ones.
  • Instruction Tuning: Smaller, task-specific models (like FLAN-T5 Large) often outperformed larger base models, suggesting that specialized reasoning is more critical than raw parameter count for complex EE tasks.

Conclusion

The DiscourseEE framework highlights a significant shift required in the field of NLP: moving beyond simple pattern matching toward true semantic understanding. While generative models offer a path forward, challenges remain regarding "hallucinations" in sensitive health contexts and the labor-intensive nature of generative annotation. Future work will likely explore Chain-of-Thought prompting to better navigate the scattered logic inherent in human discourse.

Summary updated

BibTeX
@inproceedings{sharif2024explicit,
  author = {Omar Sharif and Joseph Gatto and Madhusudan Basak and Sarah Masud Preum},
  title = {{Explicit, Implicit, and Scattered: Revisiting Event Extraction to Capture Complex Arguments}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.673},
}
Figure