TL;DR
This paper presents the HEVS-TUW team's approach to SemEval-2023 Task 8, focusing on identifying causal claims and extracting Population, Intervention, and Outcome (PIO) entities from social media text. They achieved a respectable 4th place in causal claim detection by ensembling Transformer models and rule-based methods, while their PIO extraction system, though less successful, explored deep learning and rule-based combinations.
What Problem Does This Paper Address?
Identifying causal claims and their associated components (Population, Intervention, Outcome - PIO) is crucial for various decision-making processes, especially in healthcare. Traditionally, this task has been limited to well-structured, peer-reviewed literature. However, the SemEval-2023 Task 8 challenges researchers to tackle this problem using noisy, unstructured social media data, which presents unique difficulties due to informal language, abbreviations, and spelling errors. The task is split into two parts: first, detecting spans of text that represent causal claims or related concepts, and second, extracting the PIO entities within those claims.
Methods
The HEVS-TUW team employed distinct strategies for the two subtasks.
Subtask 1: Causal Claims Detection * Ensemble Approach: They combined multiple Transformer-based models (ROBERTa, DistilBERT) fine-tuned on different aspects of the data (all entities, only questions, only personal experiences, only claims). * Weakly Supervised Question Detector: A rule-based system was used to identify potential question spans, aiding in the classification process. * Majority Voting (MV): For combining predictions from different models, a majority voting strategy was used, giving higher precedence to predictions made by at least one model over a zero-class prediction. They also incorporated a specific rule for "claim based on personal experience" entities.
Subtask 2: PIO Extraction * Deep Learning Pipeline: This involved a combination of word-level (transformer embeddings) and character-level (orthographic features) representations. These were fed into a 1D-CNN and then a fully-connected layer to generate character-level representations. * Rule-Based Module: After the deep learning component predicted PIO entities, a rule-based module used majority voting to consolidate predictions for each PIO entity.
Datasets
The primary dataset used for both training and evaluation was the SemEval-2023 Task 8 dataset, which consists of 597 English-language Reddit posts annotated for causal claims and PIO entities. This dataset is noted for its noisy, social media nature, differentiating it from previous datasets that focused on more formal medical literature.
Key Insights and Ideas
- Ensembling Transformers is Powerful: For causal claim detection, combining multiple fine-tuned Transformer models, even those trained on slightly different objectives, yielded significant improvements.
- Weak Supervision Can Help: The inclusion of a rule-based question detector, even if weakly supervised, contributed to the overall performance in Subtask 1.
- Social Media Noise is a Challenge: The paper highlights the difficulties in processing noisy social media text, where character-level features and robust representations become more critical.
- IO Tagging for Fuzzy Boundaries: The authors found that the IO (Inside-Outside) tagging scheme was more suitable than BIO (Beginning-Inside-Outside) for PIO extraction on social media data due to the often fuzzy boundaries of these entities.
- Linear Layers Outperform CRF for IO Tagging: For the IO tagging scheme, a simple linear layer was found to be more effective than a Conditional Random Field (CRF) layer, likely due to the simplified tag set and the nature of the task.
Key Results
- Subtask 1: The HEVS-TUW team achieved 4th place in causal claim detection with an F1-score of 65.77%. Their best run showed a moderate benefit from ensembling models pre-trained on independent categories.
- Subtask 2: The PIO extraction system ranked second last on the leaderboard. The system struggled particularly with identifying the "population" frame. The best F1-score for the task was 22.20% (Run 3), which was 17.91% points lower than the top-ranked approach.
Limitations and Future Work
- Subtask 2 Majority Voting: The rule-based module for PIO extraction relied on majority voting, which could be improved by considering individual model accuracies and weighting predictions accordingly. The current scheme is stringent and might miss entities if only a few models predict them.
- PIO Extraction Performance: The PIO extraction system's performance was significantly lower than desired, especially for the "population" entity. This suggests a need for further investigation and refinement of the deep learning and rule-based components.
- Ensemble Strategy: For Subtask 1, the authors suspect that the naive majority voting approach might have led to decreased performance when combining multiple strong models (Runs 3 and 4), indicating potential for more sophisticated ensemble techniques.
- Future Work: The authors suggest adapting the rule-based system to be more lenient in selecting entities, especially when dealing with cases where only one voter predicts an entity. They also implicitly point to the need for better handling of noisy social media text in PIO extraction.
Summary updated