TL;DR
This paper presents a robust system for identifying adverse drug effects (ADE) and professions in social media text, achieving competitive results in the SMM4H 2021 shared task. The authors leverage stacked heterogeneous embeddings and a BiLSTM-CRF model for Named Entity Recognition (NER), and contextualized embeddings with Neural Networks for text classification, demonstrating strong performance across English and Spanish.
What Problem Does This Paper Address?
The proliferation of social media platforms like Twitter has led to an explosion of user-generated content, offering valuable insights into public health and societal trends. However, this data presents unique challenges for analysis due to its informal nature, including misspellings, abbreviations, and data sparsity. Specifically, this paper tackles two critical tasks within the SMM4H 2021 shared task:
- Adverse Drug Effect (ADE) Detection: Identifying mentions of adverse drug reactions in English tweets. This is crucial for pharmacovigilance, allowing for early detection of potential drug side effects.
- Profession/Occupation Identification: Detecting mentions of professions and occupations in Spanish tweets. This is relevant for understanding the impact of events like the COVID-19 pandemic on vulnerable occupations and associated mental health concerns.
The core challenge lies in accurately extracting these specific entities and classifying the sentiment of tweets from noisy, informal social media text.
Methods
The authors employed distinct yet complementary approaches for Named Entity Recognition (NER) and Text Classification:
For Named Entity Recognition (NER):
- Model Architecture: A standard BiLSTM-CRF (Bidirectional Long Short-Term Memory Conditional Random Field) sequence tagger was used. This architecture is well-suited for sequence labeling tasks like NER.
- Stacked Heterogeneous Embeddings: The novelty lies in the input embeddings. They combined:
- Byte-Pair Subword Embeddings: Captures subword information, useful for handling misspellings and out-of-vocabulary words.
- fastText Subword Embeddings: Similar to Byte-Pair, it leverages subword information.
- Contextualized Word Embeddings: Utilized pre-trained embeddings from models like BERT and RoBERTa, which capture word meaning based on context.
- Linguistic Features: Incorporated features like Part-of-Speech (POS) tags, capitalization, and orthographic features to enrich the input.
- Ensemble Strategy: For NER, a majority voting mechanism was applied at the token level across three trained models (created using a bagging strategy) to make the final prediction.
For Text Classification:
- Models: Explored traditional machine learning algorithms (Logistic Regression, SVM) and Neural Networks.
- Embeddings: Focused on sentence embeddings derived from contextualized models like BERT and RoBERTa.
- Handling Imbalance: Since the datasets were imbalanced (e.g., far fewer ADE tweets than non-ADE tweets), they employed higher class weights for minority classes during training.
- Ensemble Strategy: For text classification, a straightforward majority voting at the sentence level was used for the ensemble.
Datasets
The paper utilizes datasets from the SMM4H 2021 shared task:
- Task 1a (ADE Classification): English tweets for classifying ADE mentions. The dataset is highly imbalanced, with 1235 ADE tweets out of 17385 in the training set.
- Task 1b (ADE Span Detection): English tweets for identifying ADE entities.
- Task 7a (Profession Classification): Spanish tweets for classifying profession/occupation mentions. This dataset is also imbalanced, with 1393 positive mentions out of 6000 tweets.
- Task 7b (Profession Span Detection): Spanish tweets for identifying profession/occupation entities.
The authors used a bagging strategy to create 3-fold splits of the annotated corpus for training and evaluation.
Key Insights and Ideas
- Power of Stacked Embeddings for NER: Combining various embedding types (subword, contextualized) and linguistic features significantly boosted NER performance. This suggests that a multi-faceted representation of words is crucial for accurately identifying entities in noisy text.
- Contextualized Embeddings Dominate Text Classification: Pre-trained contextualized embeddings (BERT, RoBERTa) proved highly effective for text classification tasks, outperforming traditional word embeddings. This highlights the advantage of models that understand word meaning in context.
- Ensemble Methods Improve Robustness: The use of majority voting across multiple models (trained with bagging) helped to improve the overall performance and robustness of both NER and text classification systems.
- Handling Imbalance is Crucial: The authors explicitly addressed dataset imbalance using class weighting, a common but essential technique for achieving good performance on such datasets.
Key Results
The system, under the team name MIC-NLP, achieved competitive results on the SMM4H 2021 shared task:
- NER (Test Set):
- ADE Span Detection (Task 1b): Achieved an F1-score of 0.50, which was approximately 8 points higher than the arithmetic median of participating submissions.
- Profession Span Detection (Task 7b): Achieved an F1-score of 0.824, which was 6 points higher than the arithmetic median.
- Text Classification (Test Set):
- ADE Classification (Task 1a): Achieved an F1-score of 0.46.
- Profession Classification (Task 7a): Achieved an F1-score of 0.90.
The paper also shows that their system consistently outperformed the arithmetic median across all tasks on the test set.
Limitations and Future Work
- Limitations:
- The authors acknowledge that for Task 1a (ADE Classification), their precision was lower than the arithmetic median, though this was compensated by improved recall.
- For Task 7b (Profession Span Detection), entities of type "ACTIVIDAD" and "FIGURATIVA" were ignored in the official evaluation, although the system still treated them as regular entities.
- Future Work: The authors express a desire to "improve error analysis to further enhance our NER and text classification models." This suggests a focus on understanding the types of errors their system makes and developing targeted improvements.
Summary updated