Introduction
Extracting the "temporal flow" from Electronic Health Records (EHRs) is a critical yet challenging task in clinical informatics. While unstructured clinical notes contain vital information regarding the timing and sequence of medical events, the specialized language of medicine often renders standard Natural Language Processing (NLP) tools ineffective. At SemEval-2016 Task 12 (Clinical TempEval), the ULISBOA team addressed this by adapting IBEnt, a machine learning framework originally developed for chemical entity recognition, to the clinical domain.
Methodology
The ULISBOA system utilizes a two-module approach within the IBEnt framework, combining machine learning with domain-specific heuristics.
Entity Recognition (TIMEX3 and EVENT)
The first module employs Conditional Random Field (CRF) classifiers via the Stanford NER software. To handle clinical text, the team incorporated features such as n-grams, lemmas, and "word shapes." Recognizing the limitations of standard libraries like SUTime in identifying clinical-specific terms (e.g., "peritreatment" or "post-surgery"), the authors supplemented the system with a manually curated list of approximately 200 clinical temporal terms.
Relation Extraction
The second module identifies interactions between entities using a Shallow Language Kernel, which evaluates both global and local contexts of entity pairs. The team trained four distinct CRF classifiers for specific relation types (e.g., EVENT-EVENT vs. EVENT-TIMEX3). Furthermore, heuristic rules were implemented to refine the output; for instance, events under a "HISTORY" section header were automatically linked to a "past" temporal relation.
Experimental Setup
The system was evaluated using the Clinical TempEval 2016 dataset, consisting of 439 training and 153 test clinical notes from the Mayo Clinic. The competition featured two phases:
- Phase 1: Extraction from raw text.
- Phase 2: Relation identification given "gold standard" manually annotated entities.
Results and Discussion
The ULISBOA system demonstrated strong performance, particularly in precision-oriented tasks. In Phase 2, the team adopted a "strict" approach for Run 2, requiring agreement between the CRF classifier and heuristic rules. This strategy yielded a precision of 0.823 for narrative container relations, matching the highest precision achieved in the competition.
As shown in the results table, the system performed robustly in Event Extraction (ES) with an F1-score of 0.807, close to the competition maximum of 0.903. However, Phase 1 relation extraction faced significant challenges in recall (0.009), illustrating the difficulty of identifying complex relations without pre-identified entities. The authors also noted that contextual ambiguity (e.g., "scan" as an event vs. a machine) remains a hurdle for future development.
Conclusion
The success of the ULISBOA team highlights the feasibility of pivoting specialized NLP frameworks across domains through targeted feature engineering and lexicon curation. While the system achieved top-tier precision, future work will focus on improving recall by integrating broader medical ontologies like SNOMED-CT and exploring distant learning techniques to automate rule generation, reducing the reliance on manual curation.
Summary updated