HBUT at #SMM4H 2024 Task2: Cross-lingual Few-shot Medical Entity Extraction using a Large Language Model

Yuanzhi Ke, Zhangju Yin, Xinyun Wu, Caiquan XiongACL 2024

TL;DR

This paper tackles the challenge of extracting drug and disorder mentions from social media text, which is difficult due to informal language and limited data. The authors propose a novel two-step prompting strategy for Large Language Models (LLMs) that significantly improves performance, especially for German text, by first identifying all entities and then specifically extracting medical ones.

What Problem Does This Paper Address?

Named Entity Recognition (NER) in the medical domain, particularly from social media, is a tough nut to crack. Users often discuss drugs and their adverse reactions using informal language, abbreviations, and even misspellings. Adding to the complexity, datasets for this task are typically small, making it hard for traditional models like BERT to learn effectively, especially across multiple languages. The SMM4H 2024 Task 2 specifically aimed to extract drug and disorder/body function mentions from German, French, and Japanese social media data, presenting a significant challenge for existing methods.

Methods

The researchers opted for a Large Language Model (LLM) approach, specifically using the GLM-3-Turbo model, due to the limited training data available for fine-tuning traditional models. Their core innovation lies in a two-step prompting strategy:

  1. Entity Listing: Instead of directly asking the LLM to find drug and disorder entities, the first prompt asks the model to identify all entities present in the text. This helps the LLM better understand the context and potential entities, especially in informal text.
  2. Targeted Extraction: In the second step, the LLM is prompted to extract only drug and disorder entities, but this time, it's provided with both the original text and the list of all entities identified in the first step as context. This integrated approach helps the LLM focus on the relevant entities while still benefiting from the broader context.

They also implemented a preprocessing step to handle "sensitive words" that caused errors in the GLM API, by manually checking and removing them.

Datasets

The paper utilizes datasets from the 9th Social Media Mining for Health Research and Applications Workshop and Shared Tasks (SMM4H 2024). The task involved three languages:

  • German: From a German patient forum.
  • French: Translated from the German data.
  • Japanese: From X (Twitter).

The dataset sizes are noted as relatively small, with specific numbers provided for training, validation, and test sets for each language in the Appendix.

Key Insights and Ideas

  • LLMs are powerful for low-resource NER: The paper demonstrates that LLMs, even with prompt-based approaches, can outperform traditional models when data is scarce.
  • Two-step prompting is key: The novel two-step strategy (extract all entities first, then target medical ones) proved crucial for improving performance, especially in handling informal language and abbreviations. This suggests that a broader initial entity identification helps the LLM contextualize and extract specific medical entities more accurately.
  • German performance boost: The proposed method showed a particularly strong improvement for the German language, highlighting the effectiveness of the strategy in handling specific linguistic challenges.
  • Conventional prompts are insufficient: The authors found that direct, single-step prompts for medical entity extraction yielded poor results, underscoring the need for more sophisticated prompting techniques.

Key Results

The authors' final system, using GLM-3-Turbo with their Integrated Step-by-Step Prompting (ISBS) method, achieved competitive results.

  • Overall Performance: Their system achieved an F1 score of 0.3690, outperforming the mean F1 score of 0.4291 among all participants.
  • German Performance: The system demonstrated excellent performance on the German data, with an F1 score of 0.3817, surpassing the mean F1 score of 0.2699 for German.
  • Entity Type Breakdown (German): They achieved an F1 of 0.4918 for DRUG and 0.3653 for DISORDER, but notably, 0.0000 for FUNCTION, indicating a weakness in that specific entity type.
  • Model Selection: In a local experiment, GLM-3-Turbo (0.4358 accuracy) significantly outperformed Qwen-14B (0.1538 accuracy) for this task, with Qwen-14B exhibiting tokenization issues and incorrect entity identification.

Limitations and Future Work

  • Function Entity Extraction: The system struggled to extract "function" entities, which is a limitation.
  • Sensitive Word Handling: While they addressed errors from sensitive words, they did not have time to develop a comprehensive replacement table, suggesting this as a future improvement.
  • Manual Error Correction: The manual checking of sensitive words was time-consuming and not scalable.
  • Future Work: The authors plan to complete the sensitive word replacement table and further refine their approach. They also suggest that this prompting strategy could be beneficial for other cross-lingual, few-shot NER tasks.

Summary updated

BibTeX
@inproceedings{ke2024hbut,
  author = {Yuanzhi Ke and Zhangju Yin and Xinyun Wu and Caiquan Xiong},
  title = {{HBUT at \#SMM4H 2024 Task2: Cross-lingual Few-shot Medical Entity Extraction using a Large Language Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure