TL;DR
This paper presents the BIT@UA team's approach to two tasks in the #SMM4H 2024 shared competition: identifying adverse drug events (ADEs) in tweets and classifying tweets about children's medical disorders. They achieved competitive results using pre-trained language models like RoBERTa and BERT, demonstrating the effectiveness of these models for health-related social media analysis.
What Problem Does This Paper Address?
Social media platforms like Twitter are rich sources of real-world health information, offering insights into public health trends, patient experiences, and potential safety signals. However, extracting this valuable data requires sophisticated Natural Language Processing (NLP) techniques. This paper tackles two specific challenges within this domain:
- Adverse Drug Event (ADE) Detection and Normalization: Identifying mentions of adverse drug reactions in tweets and linking them to standardized medical terminology (MedDRA). This is crucial for pharmacovigilance and understanding drug safety.
- Children's Medical Disorder Classification: Distinguishing tweets where a user reports their child having a medical disorder from tweets that simply mention a disorder. This helps in understanding parental concerns and the prevalence of specific conditions within a population.
Methods
The BIT@UA team employed a combination of pre-trained language models and traditional machine learning techniques tailored to each task.
- Task 1 (ADE Detection and Normalization):
- Named Entity Recognition (NER): They used a RoBERTa model pre-trained on PubMed abstracts and fine-tuned on medical entity datasets. For tweet classification, they simplified the standard IOB tagging scheme to an IO scheme due to the scarcity of ADE mentions.
- Entity Normalization: Detected ADE mentions were then linked to unique MedDRA identifiers. This was framed as a classification problem. They experimented with various features (token counts, TF-IDF) and machine learning models, ultimately settling on a Random Forest classifier with 10 estimators. Preprocessing included lowercasing, stop word removal, stemming, and lemmatization.
- Task 5 (Children's Medical Disorders):
- Text Representation: They explored traditional methods like Count Vectorizer and TF-IDF for text representation.
- Classification Models: Naive Bayes, SVM with a linear kernel, Logistic Regression, and XGBoost were evaluated.
- Advanced Model: A pre-trained BERT model was also fine-tuned to assess its performance against traditional methods.
Datasets
The research utilized datasets provided by the #SMM4H 2024 shared task organizers.
- Task 1: The dataset consisted of tweets annotated with ADE mentions and linked to MedDRA identifiers. It was split into training, development, and blind testing subsets. The training set had a significant imbalance, with only a small fraction of tweets containing ADE annotations.
- Task 5: The dataset comprised tweets labeled as either "Positive" (user reports a child with a disorder) or "Negative" (tweet mentions a disorder without implying it belongs to the user's child). This dataset was also split into training, development, and blind testing subsets.
Key Insights and Ideas
- Pre-trained Models Excel: Both RoBERTa (for NER) and BERT (for classification) demonstrated superior performance, highlighting their effectiveness in capturing nuanced language in the health domain.
- Data Imbalance is a Challenge: Task 1's dataset was highly imbalanced, with ADE mentions being rare. This is a common issue in real-world data and requires careful handling.
- Variability in Expression: The authors noted that the diverse ways ADEs are expressed in social media (synonyms, abbreviations, misspellings) pose a significant challenge for accurate normalization.
- Simpler NER Tagging Suffices: For Task 1, the IO tagging scheme proved adequate, suggesting that complex schemes might not always be necessary when entity mentions are scarce and non-overlapping.
Key Results
- Task 1: The BIT@UA system achieved an F1-score of 0.397 for entity recognition and 0.295 for normalization on the testing subset. While competitive, the normalization score saw a considerable drop from the development set, suggesting room for improvement.
- Task 5: The BERT model achieved the highest performance, with an F1-score of 0.840 on the blind testing subset. This was 1.8 percentage points above the mean of all submissions, though slightly below the median, indicating strong performance but also potential for further refinement.
Limitations and Future Work
- Task 1 Normalization: A key limitation identified is the system's inability to assign MedDRA identifiers not seen during training. Future work should focus on enabling the system to consult the full MedDRA terminology for comprehensive normalization. The authors also suggest exploring the BIO tagging scheme for NER and more thorough hyperparameter optimization.
- Task 5 Performance: While the BERT model performed well, the F1-score was below the median, suggesting that further improvements are possible.
- General: The authors conclude that BERT-based models are on par with the state-of-the-art for entity recognition and document classification in this domain.
Summary updated