Introduction
Clinical TempEval 2016 represents a significant advancement in automated temporal reasoning within medical records. While identifying medical events and dates has reached near-human accuracy, mapping the complex web of temporal relationships—such as determining if a symptom preceded or coincided with a treatment—remains a formidable challenge.
Unlike general "newswire" data, clinical text is characterized by shorthand, specialized jargon, and implicit temporal logic (e.g., "postoperative" relative to a surgery). This paper details a shared task competition designed to benchmark systems on extracting these critical timelines, moving toward automated clinical decision support and large-scale medical research.
Methodology and Task Design
The competition involved 14 research teams tackling nine sub-tasks categorized into three domains:
- Time Expressions: Identifying and classifying spans like "Friday" or "postoperative."
- Event Expressions: Identifying medical events (e.g., "bleeding") and attributes such as modality (actual vs. hypothetical) and polarity (negated vs. positive).
- Temporal Relations: Determining relationships between events and the document creation time, or identifying "narrative containers."
A central innovation was the "Container Philosophy." To avoid the computational explosion of linking every event to every other event, the task focused on narrative containers. This approach assumes that time expressions (like "Friday") serve as temporal "buckets" containing specific events, thereby simplifying the resulting temporal graph and making the extraction task more tractable for machine learning models.
Datasets
The evaluation utilized the THYME corpus, comprising 600 de-identified clinical and pathology notes from Mayo Clinic cancer patients. The data was split into training (50%), development (25%), and test (25%) sets. Annotations followed an extended version of TimeML, specifically adapted for clinical nuances like "Degree" (e.g., slight nausea) and "PrePostExp" (e.g., postoperative).
Key Results
Participating teams used a variety of approaches, ranging from rule-based baselines to sophisticated machine learning models like Conditional Random Fields (CRFs), Hidden Markov Model-Support Vector Machine (HMM-SVM) hybrids, and early adoptions of Recurrent Neural Networks (RNNs).
- Events and Times: Top-performing systems, such as UTHealth and UtahBMI, achieved F1 scores between 0.80 and 0.90, which is considered near-human performance.
- Temporal Relations: This remains the most difficult area. Linking events to document creation time yielded F1 scores around 0.75, while identifying narrative containers (the "hardest" task) saw top scores between 0.47 and 0.57.
- Progress: Despite the difficulty, the performance gap between AI and humans in temporal relations was halved compared to the 2015 results, indicating rapid maturation in the field.
Discussion and Future Work
The study highlighted that administrative improvements, such as streamlining Data Use Agreements (DUAs), significantly increased community participation. However, several limitations persist. The dataset's focus on a single institution and a specific condition (colon cancer) may limit generalizability. Furthermore, while systems are adept at pattern matching, they still lack the deep temporal reasoning required for complex medical narratives. Future iterations of Clinical TempEval aim to include more diverse medical conditions and multi-institutional data to improve system robustness.
Summary updated