TL;DR
This paper explores a joint training approach using BERT for classifying, extracting, and normalizing adverse drug effects (ADEs) from tweets. They found that a larger, imbalanced dataset surprisingly yielded better results than a smaller, more balanced one, and a fine-tuned BERT model achieved excellent performance on a COVID-19 symptom classification task.
What Problem Does This Paper Address?
Social media platforms like Twitter offer a rich source of real-time information for public health surveillance, particularly for identifying adverse drug effects (ADEs). However, this data is inherently noisy, with informal language, abbreviations, and a scarcity of relevant mentions. Existing approaches often treat ADE classification, extraction, and normalization as separate tasks, which can be inefficient. This paper tackles the challenge of jointly training a model to perform all three tasks simultaneously on Twitter data, aiming to improve efficiency and performance. They also address a specific subtask of classifying COVID-19 related tweets for symptoms.
Methods
The core of the approach relies on a pre-trained BERT model. For Task 1 (ADE classification, extraction, and normalization), they experimented with a joint training strategy. Key methodological components include:
- Pre-processing: Minimal pre-processing was applied to Twitter data to preserve potentially useful information within usernames and hashtags. Overlapping or nested spans were handled by selecting the longest span. A custom character embedding was created, excluding characters appearing less than 20 times in the training set.
- Bi-LSTM Hidden Layer: To further enhance the BERT representations, a bidirectional LSTM (Bi-LSTM) layer was added. This layer incorporated character embeddings and universal POS (UPOS) tags predicted by the Stanza pipeline, aiming to model interactions between these features across the entire tweet context.
- ADE Identification (Subtask 1a): The [CLS] token output from the BERT model was fed into a softmax layer for binary classification (ADE or NOADE).
- ADE Extraction & Normalization (Subtasks 1b & 1c): These tasks were approached jointly. The training data was reformulated using a BIO labeling scheme that included associated MedDRA tags. A greedy approach was used to derive the final tweet and token classifications. Spans were identified based on uninterrupted sequences of tokens receiving the same normalization tag.
- Task 6 Model: For the COVID-19 symptom classification task, a simpler approach was used: fine-tuning a BERT model with the [CLS] token output fed into a softmax layer.
Datasets
- Task 1:
- Training Data: The paper describes using a "Partial dataset" (CADEC data + training data with ADEs) and a "Full dataset" (task training data + whole CADEC dataset).
- CADEC (CSIRO Adverse Drug Event Corpus): Used to supplement the training data. This dataset has a higher proportion of ADE examples (approx. 37%) compared to the original SMM4H training data.
- Validation and Test Data: Provided by the SMM4H 2021 Shared Task.
- Task 6: The dataset for this task is described as "smaller, more balanced" than Task 1's data.
Key Insights and Ideas
- Data Imbalance Matters (and not always as expected): Counterintuitively, training on a larger, more imbalanced dataset (the "Full dataset" with ~16% ADEs) yielded significantly better results than a smaller, more balanced dataset (the "Partial dataset" with ~46% ADEs). This suggests that BERT models might be more robust to noise and imbalance than initially assumed, or that the sheer volume of data, even with more negative examples, provides a stronger learning signal.
- Joint Training Efficacy: While the joint training approach for Task 1 didn't outperform the median submission on all subtasks, it achieved a competitive normalization score, indicating the potential of this unified approach.
- Fine-tuning BERT is Powerful: For Task 6, a simple fine-tuning of a BERT model achieved very strong results, highlighting the effectiveness of pre-trained language models for specific downstream tasks even with relatively straightforward adaptation.
Key Results
- Task 1:
- Normalization (Subtask 1c): Achieved an above-average F1 score of 24%.
- Classification (Subtask 1a): Achieved an F1 score of 22%.
- Extraction (Subtask 1b): Achieved an F1 score of 37%.
- Full Dataset vs. Partial Dataset: Training on the "Full dataset" with a learning rate of 2x10⁻⁵ dramatically improved F1 scores for all subtasks compared to the "Partial dataset." For example, tweet classification F1 increased from 14.9% to 70.1% on the validation set.
- Precision vs. Recall: The final submission, trained on the full dataset, showed better precision but worse recall for all three subtasks compared to the average submission, resulting in an above-average F1 score only for normalization.
- Task 6:
- Achieved an above-average F1 score of 96%.
- Validation F1 scores were very high (98.6% and 98.3%), and the final submission beat the median of all submissions with an F1 score of 94%.
Limitations and Future Work
- Task 1 Performance: The authors acknowledge that their joint training approach did not achieve classification and extraction scores above the median submission for Task 1.
- Learning Rate Sensitivity: The experiments indicated that a learning rate of 1x10⁻⁴ was too high for the larger "Full dataset," leading to unusable results and an unstable model. This suggests careful tuning of hyperparameters is crucial.
- Dataset Imbalance: While the paper found benefits from a larger, imbalanced dataset, further investigation into the optimal balance and its impact on different subtasks could be valuable.
- Future Work: The paper doesn't explicitly detail future work, but the findings suggest exploring more sophisticated joint training architectures and further investigating the impact of data characteristics on model performance for ADE detection in social media.
Summary updated