Overview of #SMM4H 2024 – Task 2: Cross-Lingual Few-Shot Relation Extraction for Pharmacovigilance in French, German, and Japanese

Lisa Raithel, Philippe Thomas, Bhuvanesh Verma, Roland Roller, Hui-Syuan Yeh, Shuntaro Yada, Cyril Grouin, Shoko Wakamiya, Eiji Aramaki, Sebastian Möller, Pierre ZweigenbaumACL 2024

TL;DR

This paper presents Task 2 of the #SMM4H 2024 shared task, focusing on cross-lingual, few-shot relation extraction for adverse drug reactions (ADRs) from patient-generated text in French, German, and Japanese. The challenge lies in building models that can generalize from limited data to identify drugs, disorders, and their causal relationships, highlighting the difficulties of processing noisy, informal, multilingual text.

What Problem Does This Paper Address?

Pharmacovigilance, the monitoring of drug safety, is crucial for public health. While clinical trials identify many side effects, real-world patient experiences, especially those with pre-existing conditions or unique demographics, can reveal additional adverse drug reactions (ADRs). Traditional methods of collecting this information are slow and often inaccessible. Social media platforms and patient forums offer a rich, real-time source of patient-reported ADRs. However, extracting this information automatically is challenging due to the informal language, misspellings, and the need to understand complex relationships between drugs and adverse events. Furthermore, most existing resources and research are English-centric, leaving a significant gap in our ability to monitor drug safety in other languages. This paper addresses this gap by focusing on cross-lingual, few-shot relation extraction for ADRs in French, German, and Japanese.

Methods

The task involved two subtasks: Named Entity Recognition (NER) for identifying mentions of DRUG, DISORDER, or FUNCTION, and Joint Named Entity and Relation Extraction (joint NER+RE) to identify relationships between these entities, specifically "CAUSED" (drug causing a disorder) and "TREATMENT_FOR" (drug treating a disorder). The core challenge was the "few-shot" learning scenario, meaning models had to perform well with very limited annotated training data.

For their baseline, the authors employed a standard approach:

  • NER: A token classification model using pre-trained Transformer models (BERT for German, XLM-RoBERTa for Japanese and French).
  • Joint NER+RE: This involved using the predicted entities from the NER task as input for a Large Language Model (LLM), specifically Llama-3-8B-UltraMedical, with carefully crafted prompts.

Participants were encouraged to develop multi-lingual systems. The evaluation used exact match criteria for both entities and relations.

Datasets

The data for this task was a subset of the KEEPHA dataset, comprising patient-generated text from social media (X/Twitter) and patient forums in German, French, and Japanese. The German and Japanese data came from online forums and X, respectively, while the French data was a translation of German documents to overcome the lack of French patient forums. This dataset is notable for its multilingual nature and focus on patient-generated content, differing from more curated scientific literature datasets.

Key Insights and Ideas

  • LLMs struggle with structured output: While LLMs showed promise, their output often required significant post-processing to conform to the required brat format, indicating that generating structured data directly remains a challenge.
  • Few-shot learning is tough: Even with advanced models, achieving high performance with limited data proved difficult, especially for relation extraction.
  • Language-specific challenges: Detecting "FUNCTION" mentions was particularly challenging across all languages, with Japanese showing slightly better performance, possibly due to script differences.
  • Precision vs. Recall Trade-off: Submitted systems often prioritized precision (correctly identified entities/relations) over recall (finding all entities/relations), whereas the baseline system showed a more balanced approach.
  • Data scarcity is a major bottleneck: The limited size of the training data, especially for French, significantly impacted performance.

Key Results

The overall performance across all participants was modest, with the best overall NER F1 score reaching only 52.60. The baseline system, despite not being an LLM, often outperformed the submitted systems, particularly in recall. Relation extraction results were even lower, with F1 scores below 10 for most participants, highlighting the inherent difficulty of the task. Team Yseop achieved the best few-shot results for French NER, while HBUT showed strong performance in detecting DRUG mentions in French.

Limitations and Future Work

The paper acknowledges several limitations:

  • Small dataset size: The few-shot nature of the task meant limited training data, which is a significant hurdle.
  • Imbalanced relation distribution: The "CAUSED" relation was much more frequent than "TREATMENT_FOR," making it harder to learn the latter.
  • LLM output formatting: LLMs require extensive post-processing to produce the correct output format.
  • Language-specific nuances: Differences in language structure and colloquialisms present ongoing challenges.

Future work could involve exploring more sophisticated LLM prompting strategies, developing better methods for handling discontinuous entities and noisy text, and expanding the dataset size and diversity for all languages. The authors also suggest investigating the specific benefits of individual systems to gain deeper insights.

Summary updated

BibTeX
@inproceedings{raithel2024overview,
  author = {Lisa Raithel and Philippe Thomas and Bhuvanesh Verma and Roland Roller and Hui-Syuan Yeh and Shuntaro Yada and Cyril Grouin and Shoko Wakamiya and Eiji Aramaki and Sebastian Möller and Pierre Zweigenbaum},
  title = {{Overview of \#SMM4H 2024 – Task 2: Cross-Lingual Few-Shot Relation Extraction for Pharmacovigilance in French, German, and Japanese}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure