ULISBOA at SemEval-2016 Task 12: Extraction of temporal expressions, clinical events and relations using IBEnt

Márcia Barros, Andre Lamurias, Gonçalo Figueiró, Marta Antunes, Joana Teixeira, Alexandre Pinheiro, Francisco M. Couto

Introduction

Extracting the "temporal flow" from Electronic Health Records (EHRs) is a critical yet challenging task in clinical informatics. While unstructured clinical notes contain vital information regarding the timing and sequence of medical events, the specialized language of medicine often renders standard Natural Language Processing (NLP) tools ineffective. At SemEval-2016 Task 12 (Clinical TempEval), the ULISBOA team addressed this by adapting IBEnt, a machine learning framework originally developed for chemical entity recognition, to the clinical domain.

Methodology

The ULISBOA system utilizes a two-module approach within the IBEnt framework, combining machine learning with domain-specific heuristics.

Entity Recognition (TIMEX3 and EVENT)

The first module employs Conditional Random Field (CRF) classifiers via the Stanford NER software. To handle clinical text, the team incorporated features such as n-grams, lemmas, and "word shapes." Recognizing the limitations of standard libraries like SUTime in identifying clinical-specific terms (e.g., "peritreatment" or "post-surgery"), the authors supplemented the system with a manually curated list of approximately 200 clinical temporal terms.

Relation Extraction

The second module identifies interactions between entities using a Shallow Language Kernel, which evaluates both global and local contexts of entity pairs. The team trained four distinct CRF classifiers for specific relation types (e.g., EVENT-EVENT vs. EVENT-TIMEX3). Furthermore, heuristic rules were implemented to refine the output; for instance, events under a "HISTORY" section header were automatically linked to a "past" temporal relation.

Configuration of the ULISBOA system runs for Phase 1 and Phase 2.

Experimental Setup

The system was evaluated using the Clinical TempEval 2016 dataset, consisting of 439 training and 153 test clinical notes from the Mayo Clinic. The competition featured two phases:

  • Phase 1: Extraction from raw text.
  • Phase 2: Relation identification given "gold standard" manually annotated entities.

Results and Discussion

The ULISBOA system demonstrated strong performance, particularly in precision-oriented tasks. In Phase 2, the team adopted a "strict" approach for Run 2, requiring agreement between the CRF classifier and heuristic rules. This strategy yielded a precision of 0.823 for narrative container relations, matching the highest precision achieved in the competition.

Performance metrics (Precision, Recall, F1) for TIMEX3, EVENT, and RELATION extraction across different system runs.

As shown in the results table, the system performed robustly in Event Extraction (ES) with an F1-score of 0.807, close to the competition maximum of 0.903. However, Phase 1 relation extraction faced significant challenges in recall (0.009), illustrating the difficulty of identifying complex relations without pre-identified entities. The authors also noted that contextual ambiguity (e.g., "scan" as an event vs. a machine) remains a hurdle for future development.

Conclusion

The success of the ULISBOA team highlights the feasibility of pivoting specialized NLP frameworks across domains through targeted feature engineering and lexicon curation. While the system achieved top-tier precision, future work will focus on improving recall by integrating broader medical ontologies like SNOMED-CT and exploring distant learning techniques to automate rule generation, reducing the reliance on manual curation.

Summary updated

BibTeX
@inproceedings{barros2016ulisboa,
  author = {Márcia Barros and Andre Lamurias and Gonçalo Figueiró and Marta Antunes and Joana Teixeira and Alexandre Pinheiro and Francisco M. Couto},
  title = {{ULISBOA at SemEval-2016 Task 12: Extraction of temporal expressions, clinical events and relations using IBEnt}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  doi = {10.18653/v1/s16-1196},
}
Figure