Abstract
Adverse Drug Reactions (ADRs) represent a significant public health burden, necessitating efficient extraction from official drug labels. This paper describes a multi-stage pipeline developed for the TAC2017 shared task that automatically extracts ADRs and their context from Structured Product Labels (SPLs). By combining Conditional Random Fields (CRF), Bidirectional Long Short-Term Memory (BLSTM) networks, and rule-based systems into a stacked ensemble, the system achieved a top-tier F1-score of 76.00. The results demonstrate the efficacy of hybrid architectures in handling complex document structures and imbalanced medical datasets.
Introduction
Adverse Drug Reactions (ADRs) account for approximately 7% of all hospital admissions. To mitigate these risks, the U.S. FDA requires manufacturers to provide "Structured Product Labels" (SPLs). However, these documents are often composed of a complex mixture of tables, lists, and dense paragraphs, making manual information extraction slow and costly.
The primary technical challenge is Named Entity Recognition (NER). Beyond simply identifying medical conditions, a system must determine if a condition is an ADR, assess its severity, identify if it applies only to animals, or recognize if the reaction is negated (e.g., "no evidence of nausea"). This paper explores an ensemble-based approach to address these challenges within the context of the TAC2017 Adverse Drug Reaction shared task.
Methodology
The proposed system utilizes a three-stage architecture designed to handle the structural complexity of SPLs and the linguistic nuances of medical text.
1. Document Parsing
SPLs often contain critical information within tables and lists. The system "re-engineers" the document structure to identify these elements, allowing models to treat individual table cells or list items as distinct contextual units. This structural awareness is vital for disambiguating entities that might otherwise be misinterpreted in a flat text representation.
2. Word Vectorization and Features
The system employs a dual-representation strategy:
- Hand-crafted features: For the CRF models, features such as Part-of-Speech (POS) tags, dependency relations, and specialized medical lexicons are used.
- Word Embeddings: 200-dimensional Word2vec vectors were trained on a corpus combining general medical text (PubMed, PMC, Wikipedia) and unannotated drug labels provided for the task.
3. Tagging Models and Ensembles
The pipeline integrates several modeling techniques:
- Rule-based Systems: Employed for "Negation" and "Animal" classes where training data was insufficient for machine learning.
- Conditional Random Fields (CRF): Used for its strength in handling structured sequences with explicit feature engineering.
- Bidirectional LSTM (BLSTM): A deep learning model capable of capturing long-term dependencies in text.
- Stacked Ensemble: A novel configuration where the CRF's predicted class probabilities are fed into the BLSTM as additional token-level embeddings. This allows the deep learning model to benefit from the CRF's lexicon-based strengths.
Datasets
The study utilized the TAC2017 Adverse Drug Reaction shared task dataset, which consists of:
- Annotated Training Set: 101 drug labels containing 12,792 ADR mentions.
- Unannotated Set: 2,208 drug labels for unsupervised training of word embeddings.
- Test Set: An unseen subset of documents for final evaluation.
The dataset exhibited significant class imbalance. While ADR mentions were abundant (>12,000), classes like "Animal" (44 mentions) and "Negation" (95 mentions) were extremely sparse, posing a challenge for purely data-driven models.
Results and Discussion
The researchers submitted two primary runs to the TAC2017 competition. Run #1, which used a hybrid mix of Rule-based, CRF, and BLSTM models, achieved the highest F1-score of 76.00. Run #2, utilizing the Stacked CRF+BLSTM for most classes, followed closely with an F1-score of 75.58.
Key Insights
- The Stacked Advantage: Stacking CRF and BLSTM models generally outperformed individual models. The CRF excels at utilizing medical lexicons, while the BLSTM captures broader linguistic context.
- The Role of Rules: For rare classes ("Animal", "Negation"), simple dictionary-based and rule-based taggers proved more reliable than neural networks, which struggled to generalize from limited data.
- Contextual Disambiguation: Parsing document structure helped the system distinguish between relevant and irrelevant mentions. For instance, the word "serious" is only tagged as a "Severity" marker when contextually linked to a reaction.
- Simplification Trade-offs: The authors chose to ignore discontinuous annotations (e.g., split ADRs), which accounted for 6.8% of the gold standard. While this simplified the tagging scheme, it likely limited the maximum achievable recall.
Conclusion
The ensemble approach combining traditional machine learning, deep learning, and rule-based systems proved highly effective for ADR extraction in the TAC2017 task. Future work will focus on developing more informative features for CRFs, exploring knowledge-infused embeddings that incorporate medical ontologies, and investigating more sophisticated meta-classifiers for ensemble combination.
Summary updated