TL;DR
This paper investigates how to improve neural relation extraction in clinical notes by incorporating external knowledge. They found that adding semantic class information for relation triggers significantly boosts performance, while other external features like drug-problem associations or general concept co-occurrence offer more modest gains.
What Problem Does This Paper Address?
Extracting relationships between medical concepts (like treatments and problems, or tests and problems) from clinical notes is a crucial task for building automated healthcare systems and enabling research. While recent neural network approaches have shown promise, they often struggle with correctly identifying the presence of a relation (sensitivity) and distinguishing between similar relation types, leading to errors like missing relations or "hallucinating" incorrect ones. This paper aims to pinpoint these weaknesses and explore how external linguistic and domain knowledge can help overcome them.
Methods
The authors start with a Segment Convolutional Neural Network (SegCNN) model, a standard approach for this task. To understand its limitations, they perform a detailed analysis using confusion matrices, categorizing errors into poor sensitivity, hallucinated relations, and confusable relations. They then propose and evaluate three extensions:
- Drug-problem association (Drugbank): Incorporating compatibility scores between drugs and problems, and adverse drug effects, using the Drugbank ontology.
- Concept-concept association (PMI): Using Pointwise Mutual Information (PMI) to score the association between concept pairs, derived from a large corpus (MIMIC-III).
- Semantic classes: Adding features that represent semantic classes of lexical triggers (words that indicate a relation) identified in the sentence.
Datasets
The primary dataset used is the i2b2/VA challenge dataset for clinical relation extraction, specifically focusing on discharge summaries. The authors mention using a modified data split for their experiments, with 170 documents for training and 256 for testing, along with a small development set. For PMI features, they utilize the MIMIC-III corpus.
Key Insights and Ideas
- Semantic classes are powerful: Explicitly providing semantic class information for relation triggers significantly improves the model's ability to both detect relations and differentiate between them. This suggests that the explicit linguistic cues are more effective than relying solely on the dense vector representations learned by the CNN.
- Distinguishing presence vs. type: The analysis reveals that identifying whether a relation exists is often more challenging than classifying which type of relation it is, especially for treatment-problem and test-problem relations.
- PMI helps with relation presence: Pointwise Mutual Information (PMI) features seem to help the model decide if a pair of concepts should be linked, reducing the tendency to proclaim unrelated concepts as related. However, it doesn't significantly improve the distinction between different relation types.
- Drugbank coverage is limited: The Drugbank ontology, while conceptually useful, had limited coverage for the specific drug-problem pairs in the dataset, leading to only a modest impact on performance.
Key Results
- Adding semantic classes led to the largest improvement, with an almost 7-point increase in F1 score for TrP relations and a 6-point increase for TeP relations.
- PMI features showed a small positive effect for PP relations (+0.5 F1), where coverage was highest.
- The Drugbank association features had a modest improvement for TrP relations.
- Confusion matrix analysis showed that semantic classes generally increased correct classifications and sensitivity, while PMI features helped reduce hallucinated relations at the cost of some sensitivity for less frequent relations.
Limitations and Future Work
- Limited coverage of external features: The Drugbank ontology had limited coverage, and while PMI had better coverage, it still didn't cover all concept pairs.
- Sensitivity issues with less frequent relations: While semantic classes improved overall performance, they sometimes shifted counts from less frequent relations (TrIP, TrNAP, TrWP) to incorrect ones.
- Future work could explore: More comprehensive ontologies, different methods for inducing concept associations, and further investigation into how to best leverage semantic information for all relation types. The authors also make their code available for further research.
Summary updated