TL;DR
This paper details the University of Zurich's participation in the SMM4H 2018 shared task, focusing on automatic detection of drug mentions, medication intake, and adverse drug reactions from social media. They experimented with various machine learning models, including feature-based classifiers, shallow neural networks, and recurrent convolutional neural networks (RCNNs), achieving competitive results.
What Problem Does This Paper Address?
The paper addresses the challenge of automatically extracting health-related information from noisy and informal social media text, specifically focusing on drug-related posts. The SMM4H 2018 shared task aimed to develop systems for tasks like identifying drug names, classifying medication intake, and detecting adverse drug reactions. This is crucial for public health surveillance, pharmacovigilance, and understanding patient experiences with medications. The inherent challenges include misspellings, abbreviations, slang, and the informal nature of social media language.
Methods
The authors explored a range of machine learning approaches:
- Feature-based Classifiers: They used traditional methods like Logistic Regression, Support Vector Machines (SVMs), and Naive Bayes. These systems relied on features such as bag-of-lemmas, pre-trained word embeddings, and custom lists of drug-related terms.
- Shallow Neural Network with Tunable Embeddings: A simple end-to-end learning system using a shallow neural network with tunable pre-trained embeddings (fastText) fine-tuned on the task data.
- Recurrent Convolutional Neural Network (RCNN): For drug name detection (Task 1), they employed an RCNN that combines recurrent structures (GRU) to capture context with convolutional layers and max-pooling for relevance. They experimented with both word-based and character-based RCNNs, finding the word-based version to be superior. A key innovation was augmenting word embeddings with a binary feature indicating if a word was present in the RX Norm drug name list.
- Ensemble of Convolutional Neural Networks (CNNs): For adverse drug reaction detection (Task 3), they built a tiered ensemble of CNNs, using a majority class downsampling strategy to handle class imbalance.
Preprocessing steps were extensive and included URL splitting, camel case splitting, hashtag stripping, expansion of abbreviations like "w/", splitting at punctuation, and replacing numbers/usernames with placeholders. They also utilized SpaCy for lemmatization and part-of-speech tagging.
Datasets
The primary datasets used were provided by the SMM4H 2018 shared task for three sub-tasks:
- Task 1: Drug name detection.
- Task 2: Medication intake classification (personal, possible, non-intake).
- Task 3: Adverse drug reaction detection.
The paper notes that not all tweets were available at the time of participation, and they split the available data into training and development sets. They also mention using external resources like RX Norm for drug name lists and Google Books Ngram Corpus for filtering common words.
Key Insights and Ideas
- The Power of Domain-Specific Features: Augmenting general word embeddings with domain-specific features, such as whether a word is a known drug name (from RX Norm), significantly improved performance, especially for the RCNN model.
- Handling Noisy Social Media Data: The extensive preprocessing steps, including handling URLs, camel case, and replacing sensitive information, were crucial for making the noisy social media text amenable to machine learning models.
- RCNN Effectiveness for Text Classification: The Recurrent Convolutional Neural Network architecture proved effective for capturing contextual information and local features in tweets, leading to strong results in drug name detection.
- Challenges in Nuance and Ambiguity: The error analysis revealed that distinguishing between endogenous substances and supplements, or correctly interpreting words like "vitamin" or "pills" in context, remained challenging. Misspellings and ambiguous terms also posed difficulties.
- Ensembling for Imbalanced Data: For imbalanced tasks like ADR detection, ensembling CNNs with a downsampling strategy was explored, though it faced challenges in achieving optimal results.
Key Results
- Task 1 (Drug Name Detection): The word-based RCNN achieved an F1 score of 0.92 on the development set, outperforming a baseline biLSTM model (F1 score of 0.881).
- Task 2 (Medication Intake): The best feature-based system (linear SVM) achieved a micro-averaged F1 score of 0.713 on the development set.
- Task 3 (ADR Detection): The best official score achieved was an F1 score of 0.522. The ensemble CNN approach showed promising results on the development set (F: 48.3) but performed poorly in the official evaluation, possibly due to configuration errors.
- Official Scores: Table 4 provides official results, with their best submission for Task 1 achieving an F1 of 0.880, Task 2 an F1 of 0.713, and Task 3 an F1 of 0.522.
Limitations and Future Work
- Data Availability: The incomplete availability of tweets from the original dataset was a limitation.
- Configuration Errors: The authors suspect a bug or configuration error in their ensemble CNN system for Task 3, which led to unexpectedly poor official results.
- Distinguishing Nuances: The difficulty in distinguishing between endogenous substances and supplements, and correctly interpreting ambiguous terms, remains a challenge.
- Future Work: The authors plan to continue experimenting with the provided datasets and report new results at future workshops, indicating an ongoing interest in this research area. They also mention considering a combination of character-based and word-based RCNNs, which they couldn't implement due to time constraints.
Summary updated