Abstract
Extracting temporal information from clinical narratives is essential for longitudinal patient analysis. This paper describes the UTHealth system for the SemEval-2016 Clinical TempEval challenge. The system utilizes a modular pipeline combining HMM-SVM sequence tagging, LIBLINEAR SVM classifiers, and transitive closure for relationship mapping. The approach achieved top performance across all six sub-tasks, demonstrating the effectiveness of integrating traditional linguistic features with modern word representations and domain-specific heuristics.
Introduction
Electronic Health Records (EHRs) contain vast amounts of unstructured data where critical temporal information—such as the timing and duration of medical events—is often implicit. Automatically extracting this "timeline" is challenging due to the complexity of medical language and the density of temporal relations.
The 2016 Clinical TempEval (SemEval-2016 Task 12) benchmarked systems on six sub-tasks: identifying event and time spans, determining their attributes, relating events to document creation time (DocTimeRel), and identifying narrative container relations (TLINK:Contains).
Methodology
The UTHealth system employs a four-stage pipeline designed to process raw clinical notes:
- Span Recognition: Event and time expressions are identified using an HMM-SVM sequence tagger. Features include lexical n-grams, POS tags, Brown clustering, and word embeddings trained on the MIMIC-II corpus. The rule-based SUTime tagger is also integrated as a feature.
- Attribute Identification: SVM classifiers determine properties like event polarity (occurrence vs. non-occurrence) and time expression types.
- DocTimeRel Classification: A LIBLINEAR SVM relates events to the Document Creation Time (BEFORE, AFTER, OVERLAP, or BEFORE/OVERLAP).
- Relation Extraction (TLINK:Contains): This task identifies if one temporal entity "contains" another. The team used transitive closure (Floyd-Warshall algorithm) to expand the training set and heuristic filtering to manage the search space of potential pairs.
Results
The system was evaluated on the THYME corpus of clinical notes from the Mayo Clinic. It secured first place in all six sub-tasks when processing raw text (Phase 1).
As shown in the table above, the system achieved high F1-scores for span identification, particularly for events (0.903). The performance on DocTimeRel (F1: 0.756) and the challenging TLINK:Contains task (F1: 0.479) also led the field, despite the inherent difficulty of relation extraction in clinical text.
Analysis and Key Insights
A significant factor in the system's success was the use of transitive closure. By inferring implicit relations (e.g., if A contains B and B contains C, then A contains C), the researchers provided the models with a much richer training signal for the sparse "Contains" relations.
Furthermore, the system's performance varied significantly across different temporal relation values, reflecting the distribution of the training data.
The system was most accurate at identifying OVERLAP relations (90.1%), which were the most frequent in the training data. In contrast, the rare BEFORE/OVERLAP category proved much harder to classify, yielding only 51.5% accuracy. The team also noted that section awareness (e.g., distinguishing "History" from "Evaluation") was a critical feature for narrowing down likely temporal relations.
Conclusion
The UTHealth system demonstrates that a well-engineered pipeline combining machine learning with domain-specific heuristics and data augmentation techniques like transitive closure can effectively extract temporal information from clinical text. While the system excelled at span and attribute identification, future work will address remaining challenges in preposition handling and the identification of complex event-event relations across different sections of a clinical record.
Summary updated