TL;DR
This paper presents a novel neural transition-based joint model for simultaneously extracting adverse drug effect (ADE) mentions from tweets and normalizing them to standardized MedDRA terms. Their system achieved competitive results in the SMM4H 2021 shared task, outperforming the average F1 score by 1-2%.
What Problem Does This Paper Address?
Social media platforms like Twitter have become a rich source of real-world health information, including discussions about adverse drug effects (ADEs). Mining this information is crucial for pharmaceutical companies to monitor drug safety and for public health surveillance. However, extracting and standardizing these ADE mentions from noisy, informal text like tweets is challenging. Specifically, the task involves two sub-problems: first, identifying the exact text spans in a tweet that describe an ADE (extraction), and second, mapping these identified mentions to a canonical term in a medical knowledge base like MedDRA (normalization). This paper tackles the joint problem of performing both extraction and normalization efficiently and accurately.
Methods
The authors propose a neural transition-based joint model that frames the entire task as a sequence labeling problem. This approach leverages a transition system, similar to those used in named entity recognition (NER), to incrementally build the output. The model uses a set of actions: SHIFT (move to the next token), OUT (ignore the current token), REDUCE (complete an entity), and SEGMENT (segment entities). Crucially, they extend this by introducing LINKING actions to perform the normalization step simultaneously with extraction.
The input representation for each token is a concatenation of three embeddings:
- Character-level word representation: Learned using a CNN.
- Non-contextual word representation: Initialized with GloVe embeddings.
- Contextual word representation: Initialized with ELMo embeddings.
For efficient decoding, they employ a greedy search algorithm to minimize the negative log-likelihood of the local action classifier.
Datasets
The paper uses data from the sixth Social Media Mining for Health Applications (SMM4H) shared task in 2021, specifically subtask 1c. This subtask focuses on extracting ADE mentions from tweets and normalizing them to MedDRA preferred term IDs. The dataset is divided into training, development, and testing sets. The authors used the training and development sets for system building and submitted predictions on the testing set. The knowledge base used for normalization is MedDRA v21.1, which contains 25,463 unique preferred term IDs.
Key Insights and Ideas
- Joint Extraction and Normalization: The core innovation is the simultaneous handling of both ADE mention extraction and normalization within a single transition-based model. This avoids the potential error propagation that can occur when these tasks are treated sequentially.
- Leveraging Rich Embeddings: The combination of character-level, non-contextual (GloVe), and contextual (ELMo) embeddings provides a robust representation for tweet tokens, capturing both lexical and semantic nuances.
- Transition-based Framework: Adapting a transition-based approach, commonly used for NER, to include normalization actions offers an efficient and structured way to process the text and build the desired output.
Key Results
The authors submitted two results:
- Submission 1: A single model using the proposed neural transition-based joint model.
- Submission 2: A voting result from the top 5 best single models.
Both submissions outperformed the average F1 score of all participating teams by 1-2%. The paper provides tables showing Precision, Recall, and F1 scores for both development and test sets, indicating competitive performance. For instance, on the test set, their best submission achieved an F1 of 0.240, compared to the average F1 of 0.220.
Limitations and Future Work
- Limitations: The paper notes that the evaluation metric considers spans that overlap entirely or partially. This means partial matches contribute to the score, which might be a limitation depending on the desired strictness of extraction.
- Future Work: The authors plan to further tune the model and explore other popular contextual word representations, such as those learned from BERT, to potentially improve performance.
Summary updated