TL;DR
This paper explores how to improve the interpretability of medical code classification models by incorporating knowledge graphs (KGs) into the input representation. They found that using pre-trained KG embeddings (KGE) enhances model attention and reasoning, while explicit graph neural network (GNN) approaches for encoding domain knowledge showed limitations.
What Problem Does This Paper Address?
Assigning the correct International Classification of Diseases (ICD) codes to clinical texts is crucial for healthcare but challenging. This task, often framed as multi-label classification, is complicated by the need for extensive domain knowledge, long and complex clinical notes, and a vast, imbalanced label space. While deep learning models have shown promise, understanding why they make certain predictions is vital for trust and error analysis in healthcare. Existing work has explored incorporating external knowledge, often through KGs, but the focus has primarily been on the output (label) space or using KGs indirectly. This paper specifically investigates how to leverage graph structures within the input document space to improve both performance and, importantly, the interpretability of attention-based reasoning.
Methods
The researchers propose and evaluate two main approaches for incorporating graph structures into the input representation:
- Implicit Graph Structures (Knowledge Graph Embeddings - KGE): They represent clinical concepts (extracted as Concept Unique Identifiers or CUIs from the UMLS Metathesaurus) using pre-trained KGEs. This implicitly captures relational information from the UMLS knowledge base. They compare this to standard Word2Vec (W2V) embeddings for CUIs.
- Explicit Graph Structures (Graph Neural Networks - GNN): They use Graph Convolutional Networks (GCNs) to explicitly encode the relationships between concepts within a document. They explore two ways of constructing these graphs:
- GCNBASE: Edges are based on general KG relations from UMLS.
- GCNEHR: Edges are guided by clinical reasoning steps, derived from manual annotation and UMLS semantic information, aiming to reflect how diagnoses relate to treatments.
These graph-enhanced representations are then fed into an encoder, specifically the Label Attention Encoder (LAAT) model (a strong non-GNN baseline) and a GCN encoder, to predict ICD codes.
Datasets
The experiments were conducted on the MIMIC-III dataset, specifically using the discharge summaries and their associated ICD-9 codes. They used both the "Top-50" and "Full" versions of the dataset partitions for evaluation.
Key Insights and Ideas
- KGEs enhance interpretability: Pre-trained KGEs for concept tokens provide a more clinically intuitive attention-based reasoning compared to standard W2V embeddings. This means the model's attention is more likely to focus on clinically relevant parts of the text when using KGEs.
- Explicit GNNs have limitations: While GCNs can explicitly encode relational information, their performance in this task was overshadowed by a drop in accuracy compared to the LAAT model. This suggests that simply encoding graph structures might not be sufficient without careful consideration of how those structures are built.
- Attention mechanism struggles with negation: The attention mechanism, regardless of the input feature type, tends to ignore negation. This can lead to incorrect predictions, as clinicians often use negation to rule out conditions.
- Concept extraction is crucial: The accuracy of concept extraction (identifying CUIs) directly impacts the model's ability to leverage domain knowledge effectively. Extraneous or irrelevant CUIs can lead to noisy attention patterns.
Key Results
- KGEs improve LAAT performance: On the LAAT model, KGEs consistently showed improvements over W2V embeddings across most metrics (Precision, Recall, F1, AUC) for both Top-50 and Full dataset versions.
- GCN performance lags: The GCN encoder, even with KGE node embeddings, generally performed worse than the LAAT model. While GCNEHR (domain-knowledge guided) showed some benefits over GCNBASE, the overall performance was lower.
- Concept features vs. Text features: Concept-based features (using CUIs) generally resulted in lower performance than text-based input features, though the interpretability advantage of concept-based features might justify this trade-off in critical applications.
- Thresholding for GCNEHR: For the GCNEHR approach, optimal conditional probability thresholds for edge creation were found to be 0.7 for Top-50 and 0.5 for the Full dataset.
- Runtime efficiency: CUI input models (W2V and KGE) showed significantly faster training runtimes compared to text-input models, making them more computationally efficient.
Limitations and Future Work
- GNN limitations: The study highlights limitations of GCNs as standalone encoders, suggesting that their performance is impacted by graph construction approaches and that heuristics based on clinical reasoning might not always be optimal for non-clinical purposes.
- Dataset and version scope: Experiments were limited to MIMIC-III and ICD-9 codes. Future work could explore newer datasets like MIMIC-IV and ICD-10.
- Qualitative analysis sample size: The qualitative analysis was based on a small sample size and a single clinician's annotation. Expanding this with more data and multiple experts is suggested.
- Negation handling: The paper points out the need for better handling of negation, suggesting that incorporating negation-related concepts or improving concept extraction to capture negation could be a future direction.
- Concept descriptions: Utilizing CUI descriptions instead of just CUIs as features is proposed as a future avenue.
- Generative models and KG expansion: Leveraging generative models for KG expansion and using concept-based KGE or GCNs to augment text-based features are suggested for future research.
- Hierarchical evaluation: The paper notes that standard MLC metrics might not be ideal for hierarchical ICD codes and suggests exploring hierarchical evaluation metrics.
Summary updated