Team Yseop at #SMM4H 2024: Multilingual Pharmacovigilance Named Entity Recognition and Relation Extraction

Anubhav GuptaACL 2024

TL;DR

This paper details Yseop's participation in the SMM4H 2024 shared tasks, focusing on multilingual pharmacovigilance. They developed RoBERTa-based systems for extracting adverse drug events (ADEs) from English tweets and identifying named entities and relations in French and Japanese, achieving strong results, particularly in the French and Japanese tasks.

What Problem Does This Paper Address?

The paper addresses the critical challenge of pharmacovigilance, the science and activities relating to the detection, assessment, understanding, and prevention of adverse effects or any other medicine-related problem. With the rise of social media, a vast amount of publicly available health-related information is shared online. However, extracting structured, actionable insights from this unstructured text, especially across multiple languages, is a significant hurdle. Specifically, the SMM4H 2024 workshop aimed to advance the automated extraction of adverse drug events (ADEs) and their associated entities and relations from social media data, a crucial step in strengthening public health monitoring systems.

Methods

Yseop employed a RoBERTa-based approach for their systems. For the English ADE extraction task (Task 1), they fine-tuned a RoBERTa model for token classification. To improve performance, they experimented with data augmentation using English texts from a previous sub-task. For the multilingual Named Entity Recognition (NER) and Relation Extraction (RE) tasks (Task 2), they developed separate models for French and Japanese. The French model utilized a pre-trained model adapted to the clinical domain (DrBERT-CASM2) and a large language model (Mistral LLM) for prompt-based extraction. The Japanese NER model was fine-tuned on a domain-specific RoBERTa model (daisaku-s/medtxt_ner_roberta). For Japanese relation extraction (Task 2b), they framed it as a sequence classification task.

Datasets

The paper utilized datasets from the SMM4H 2024 workshop.

  • Task 1 (English ADE Extraction): The training set comprised 17,385 tweets, with 1,239 containing ADEs. They also used data from the SM-ADE sub-task (Wakamiya et al., 2023) for augmentation.
  • Task 2 (Multilingual NER & RE):
    • French: A small dataset of 4 documents.
  • Japanese: A larger dataset of 392 documents.
  • The data was annotated in the brat standoff format.

Key Insights and Ideas

  • Domain Adaptation is Key: The French and Japanese systems, which leveraged models pre-trained or adapted to the clinical domain, significantly outperformed the English system, highlighting the importance of domain-specific knowledge for pharmacovigilance tasks.
  • Data Augmentation Benefits: While data augmentation was explored for the English task, the paper notes that the augmented model did not show significant performance improvement over the model trained on the original data. This suggests that the effectiveness of augmentation can be task- and data-dependent.
  • Tokenization Challenges in Japanese: The paper identifies tokenization errors as a significant factor contributing to the poor performance of the Japanese NER model. Specific examples illustrate how the tokenizer incorrectly segments words, leading to missed entities.
  • Relation Extraction Complexity: The Japanese relation extraction task (Task 2b) proved particularly challenging, with the winning team achieving a very low overall F1 score, indicating the inherent difficulty of this task in low-resource settings.

Key Results

  • Task 1 (English): Achieved an F1-norm score of 40.0, placing them in the top 50 percentile. The F1-NER score was higher at 47.2.
  • Task 2a (French NER): The French model achieved an F1 score of 0.4132.
  • Task 2a (Japanese NER): The Japanese NER model achieved an F1 score of 0.5827.
  • Task 2b (Japanese RE): The Japanese relation extraction model achieved an F1 score of 0.0301.
  • The French and Japanese systems were noted as the best performing systems for Task 2.

Limitations and Future Work

  • Task 1 Dataset Error: Yseop acknowledges that they performed Task 1 on the SMM4H-2023 Task 5 dataset due to an error on their part, which might have impacted their performance compared to using the intended SMM4H 2024 dataset.
  • Japanese Tokenization: The paper explicitly mentions the tokenization error in Japanese as a limitation for the NER task and suggests this as a reason for the suboptimal performance.
  • Future Work: The authors plan to explore approaches like GLiNER and XMC to improve NER performance in Task 1 and Task 2a. They also intend to investigate further methods for relation extraction.

Summary updated

BibTeX
@inproceedings{gupta2024team,
  author = {Anubhav Gupta},
  title = {{Team Yseop at \#SMM4H 2024: Multilingual Pharmacovigilance Named Entity Recognition and Relation Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure