TL;DR
This paper tackles the challenge of automatically identifying and normalizing mentions of adverse drug reactions (ADRs) in informal social media posts to standardized medical codes (MEDDRA). The authors propose a novel few-shot learning approach that leverages pre-trained word embeddings and external medical knowledge to overcome the scarcity of labeled data, outperforming the average participant in the SMM4H shared task.
What Problem Does This Paper Address?
Social media platforms like Twitter are a rich source of real-world information, including potential adverse drug reactions (ADRs). However, extracting and standardizing this information for medical research is a significant challenge. The informal language, abbreviations, and misspellings common in social media posts create a large gap between user-generated content and the formal medical terminology required for analysis. Furthermore, creating large, annotated datasets for training machine learning models is expensive and time-consuming, leading to a scarcity of labeled data, especially for less common ADRs. This paper addresses the problem of automatically identifying ADR mentions in tweets and normalizing them to their corresponding Medical Dictionary for Regulatory Activities (MEDDRA) codes, a crucial step for pharmacovigilance and drug safety monitoring.
Methods
The team's system for the SMM4H Shared Task 3 involved three steps: ADR mention classification, ADR extraction, and ADR normalization. They adopted existing state-of-the-art methods for the first two steps. Their primary contribution lies in the third step: ADR normalization, which they frame as a Few-Shot Learning (FSL) problem.
Their FSL approach involves creating an embedding-based classifier. This classifier compares the vector representation of an extracted ADR mention to its nearest neighbors in a vector space. This space is populated with:
- Vector representations of ADR mentions from the training data.
- Vector representations of prior medical knowledge.
To generate these vector representations, they utilize the pre-trained Google News Word2Vec model. For their prior knowledge, they leverage the 319 MEDDRA Preferred Terms (PTs) present in the training data, combining their preferred names from the MEDDRA vocabulary and the Consumer Health Vocabulary (CHV) mapped via UMLS. This prior knowledge dataset contains 1,854 preferred names.
At test time, an unseen ADR mention is also converted into a vector using Word2Vec. A 1-Nearest Neighbor (1-NN) search with cosine similarity is then performed to find the most similar concept in the combined training and prior knowledge embeddings, thereby normalizing the ADR mention to its MEDDRA code. They also experimented with ensemble methods combining a Recurrent Neural Network (RNN) with their FSL approach.
Datasets
The paper utilizes datasets provided for the 2019 Social Media Mining for Health Applications (SMM4H) Shared Task 3. These datasets contain tweets with and without ADR mentions. For subtask 3, the training data consists of 1212 ADR mentions mapped to 319 MEDDRA codes after preprocessing. The authors note that a significant portion of these codes have very few training samples (46.7% have only one sample, and 79.6% have less than five). They also incorporate external data from the UMLS and the MEDDRA vocabulary as prior knowledge.
Key Insights and Ideas
- The Power of Few-Shot Learning for Scarce Data: The core insight is that FSL can effectively handle the challenge of limited labeled data in medical text normalization, a common issue in this domain.
- Leveraging Prior Medical Knowledge: Integrating external medical knowledge (from UMLS and MEDDRA) alongside training data significantly boosts performance, especially for rare ADRs. This "prior knowledge" acts as a rich source of semantic information.
- Embedding-Based Similarity for Normalization: Using word embeddings and a nearest-neighbor approach allows for robust normalization by capturing semantic similarity between informal mentions and formal medical codes.
- Complementary Strengths of RNNs and FSL: While RNNs perform well with abundant data, FSL excels with sparse data. The paper explores ensembles, suggesting that combining these approaches could yield even better results, though their specific ensemble configurations didn't outperform pure FSL on the test set.
Key Results
The authors' FSL approach achieved a relaxed F-score of 0.345 and a strict F-score of 0.244 on the test data. These scores outperform the average performance of all participants in Task 3 (Task 3 AVG: 0.297 relaxed F-score, 0.211 strict F-score). This demonstrates the effectiveness of their few-shot learning strategy in a real-world shared task setting. Their ensemble methods, while performing well on development data, did not generalize as well to the test set.
Limitations and Future Work
- Generalization of Ensembles: The ensemble methods (RNN+FSL) did not generalize as well to the test data as the standalone FSL approach, indicating potential overfitting or issues with confidence estimation in disagreement scenarios.
- Discrimination of Rare vs. Common Cases: The authors plan to focus future work on better discriminating between "commonly seen cases" (classes with sufficient training data) and "rare cases" (classes with insufficient training data). This will allow for a more sophisticated combination of deep neural networks and few-shot learning.
- Improving Robustness: While the FSL approach shows promise, further work is needed to make it even more robust and efficient for linking ADR tweet mentions to MEDDRA codes.
Summary updated