Introduction
Traditional Event Extraction (EE) methodologies have long relied on a "span-extraction" paradigm, where models identify events and highlight contiguous strings of text to fill argument roles. While effective for structured news reporting, this approach falters in the nuanced, informal landscape of social media. Researchers from Dartmouth College address this gap by introducing DiscourseEE, a framework that treats argument extraction as a generative task, enabling the capture of information that is not explicitly stated or is distributed across a document.
Redefining Argument Extraction
The core challenge in informal discourse, such as online health forums, is that critical details are often non-explicit. The authors categorize arguments into three distinct types:
- Explicit: Contiguous text spans (e.g., "prostate cancer").
- Implicit: Information that must be inferred from context (e.g., a goal dosage of "0mg" for a patient attempting to quit a medication).
- Scattered: Details that must be synthesized from multiple parts of a post or an entire comment thread.
The DiscourseEE Dataset
To benchmark this new approach, the researchers developed the DiscourseEE dataset, sourced from Reddit communities focused on Opioid Use Disorder (OUD). The dataset contains 7,464 argument annotations across three event types: Taking MOUD, Return to Usage, and Tapering.
Significantly, the dataset reveals that the vast majority of information in these peer-to-peer interactions is non-explicit: 51.2% of arguments are implicit and 17.4% are scattered. This complexity renders traditional extractive models, which achieved only a 17.13 RM_F1 score, largely obsolete for this domain.
Generative Modeling and Evaluation
The proposed framework shifts from "finding the span" to "generating the answer." By utilizing Large Language Models (LLMs) and instruction-tuned models like FLAN-T5, the system can describe arguments in natural language.
Because generative models may use synonyms (e.g., "runny nose" vs. "nasal discharge"), the authors implemented Relaxed Matching (RM_F1). This metric uses BERT embeddings to calculate semantic similarity, considering a response correct if it meets a 0.75 similarity threshold with the ground truth. This shift more accurately reflects model performance compared to rigid "Exact Match" standards.
Key Findings
- The Hidden Majority: Nearly 70% of vital health information is missed by traditional extractive models because it is not written verbatim.
- Model Performance: GPT-4 emerged as the top performer with a 41.98 RM_F1 score. However, even advanced LLMs struggle with implicit reasoning, achieving only 36.53% recall for implicit arguments compared to 55.14% for explicit ones.
- Instruction Tuning: Smaller, task-specific models (like FLAN-T5 Large) often outperformed larger base models, suggesting that specialized reasoning is more critical than raw parameter count for complex EE tasks.
Conclusion
The DiscourseEE framework highlights a significant shift required in the field of NLP: moving beyond simple pattern matching toward true semantic understanding. While generative models offer a path forward, challenges remain regarding "hallucinations" in sensitive health contexts and the labor-intensive nature of generative annotation. Future work will likely explore Chain-of-Thought prompting to better navigate the scattered logic inherent in human discourse.
Summary updated