TL;DR
This paper explores two approaches for extracting temporal information from clinical notes: a standard Recurrent Neural Network (RNN) and a more sophisticated joint inference system built with DeepDive. The joint inference approach, which leverages structured prediction, significantly outperformed the vanilla RNN for identifying time and event mentions, and showed promise for relating events to document creation times, despite limitations in training data.
What Problem Does This Paper Address?
Extracting temporal information from clinical text is crucial for understanding patient disease progression, enabling medical reasoning, and supporting longitudinal research. While electronic health records (EHRs) contain structured timestamps, a wealth of temporal information is embedded within unstructured clinical notes. This paper tackles the challenge of automatically identifying these temporal expressions (like dates and events) and understanding their relationships within the clinical narrative, specifically focusing on two sub-tasks: identifying temporal entities and determining the relationship between clinical events and the document's creation time.
Methods
The authors submitted two systems to SemEval-2016 Task 12: Clinical TempEval.
For Phase 1 (Temporal Entity Extraction), they compared two main methods:
- Vanilla Recurrent Neural Network (RNN): This standard approach uses a recurrent neural network with word embeddings. The RNN incorporates a context layer to model sequence history. Word embeddings were trained on large clinical corpora (MIMIC-III and UIHC) using word2vec.
- DeepDive (Joint Inference): This system leverages structured prediction, building a probabilistic graphical model. It uses a combination of logistic regression, conditional random fields (CRFs), and skip-chain CRFs. DeepDive utilizes a feature library (ddlib) to generate common textual features like context windows and dictionary lookups.
For Phase 2 (Document Creation Time Relations), they again used DeepDive. This involved:
- Date Canonicalization: For simple date mentions, they used regular expressions and heuristics to convert them into universal timestamps.
- Distant Supervision: For more ambiguous temporal phrases, they employed a distant supervision approach. This involved learning dominant temporal associations from training data (e.g., "currently" often appearing near events that overlap the document creation time).
- Inference Rules: They defined logistic regression rules to predict document-relative time values and a linear skip-chain factor to link EVENT mentions to their nearest TIMEX3 neighbors.
Datasets
- Training and Evaluation Data: The THYME corpus, consisting of clinical and pathology notes from the Mayo Clinic, was used for training and evaluation.
- Word Embedding Corpora:
- MIMIC-III database: ~2.4M notes from critical care patients.
- University of Iowa Hospitals and Clinics (UIHC) corpus: ~15M predominantly inpatient notes.
Key Insights and Ideas
- Structured Prediction Outperforms Vanilla RNNs: The paper strongly suggests that a joint inference approach (DeepDive) is more effective than a standard RNN for temporal entity extraction in clinical text, especially when training data is limited. This highlights the benefit of explicitly encoding domain knowledge and relationships.
- The Power of Domain-Specific Embeddings: Pre-training word embeddings on large clinical corpora significantly improved RNN performance, demonstrating that these embeddings capture crucial syntactic and semantic information relevant to the clinical domain.
- DeepDive's Efficiency: The DeepDive system achieved competitive results with relatively little upfront engineering (around a week), showcasing its ability to quickly encode domain knowledge into a probabilistic model.
- Feature Engineering Still Matters (to an extent): While DeepDive's feature generator libraries were powerful, the paper found that simple features like context windows and dictionary lookups were sufficient to achieve median system performance for Phase 2. POS tag features, surprisingly, had no significant impact.
- Limitations of Vanilla RNNs: The paper acknowledges that vanilla RNNs often struggle with vanishing/exploding gradients, which can hinder their performance in sequence labeling tasks compared to more advanced architectures like LSTMs.
Key Results
- Phase 1 (TIMEX3 Spans): The best DeepDive runs achieved an F1 score of 0.730, performing as well as the previous year's best system but falling short of the winning system (0.795). The RNN ensemble achieved an F1 of 0.706.
- Phase 1 (EVENT Spans): RNN models showed higher recall, while DeepDive had higher precision. Both approaches scored below the best system (0.885 vs. 0.903). The best DeepDive run achieved an F1 of 0.883.
- Phase 2 (Document Creation Time Relations): The DeepDive system achieved better performance than the median submission but fell short of the state-of-the-art. The paper does not provide specific F1 scores for this phase but indicates the system's performance was competitive.
Limitations and Future Work
- Limited Training Data: The authors explicitly state that the small scale of training data for Phase 2 limited the gains achieved. This is a significant factor impacting performance.
- Vanilla RNN Limitations: The paper points out that vanilla RNNs are susceptible to vanishing/exploding gradient problems, suggesting that more advanced recurrent architectures (LSTMs, GRUs, Tree-LSTMs) could offer better performance.
- Future Work: The authors suggest that future work should focus on closing the performance gap with state-of-the-art systems. This could involve exploring more sophisticated recurrent architectures, leveraging larger unlabeled datasets, and potentially developing more advanced feature engineering or inference strategies within the DeepDive framework. They also note that a larger collection of unlabeled notes could lead to further improvements in their document creation time relation extraction system.
Summary updated