KULeuven-LIIR at SemEval 2016 Task 12: Detecting Narrative Containment in Clinical Records

Artuur Leeuwenberg, Marie-Francine MoensSemEval @ NAACL 2016 · OpenReview

TL;DR

This paper details KU Leuven-LIIR's participation in the SemEval 2016 Task 12, focusing on extracting "narrative containment" relations in clinical text. They adapted an existing system (cTAKES-Temporal) and explored new features, finding that while their system performed above the median, significant challenges remain in handling cross-sentence relations and unknown terms.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically identifying relationships between events and temporal expressions within clinical narratives. Specifically, they focus on "narrative containment," which describes how one event or temporal expression contains another. This is crucial for understanding the temporal flow of patient histories, aiding in tasks like clinical trial matching, patient summarization, and retrospective analysis. Existing systems often struggle with the nuances of clinical text, such as long-distance dependencies, negations, and domain-specific terminology.

Methods

The KU Leuven-LIIR team built upon the existing cTAKES-Temporal system, an open-source information extraction framework for clinical text. Their approach involved two main stages:

  • Run 1: Utilized the cTAKES-Temporal system with its pre-existing features. This system employs two Support Vector Machine (SVM) classifiers: one for event-event (EE) containment and another for temporal expression-event (TE) containment.
  • Run 2: Introduced new features based on an error analysis of Run 1. These included:
  • Modified Dependency Path (MDP): Incorporating part-of-speech tags and actual words for certain token types in the dependency path between entities.
  • In-Between Sequence (IBS): Capturing the sequence of words (or their POS tags) between the two related entities.

Datasets

The primary dataset used for this task was the THYME corpus, specifically clinical records related to colon cancer. The authors participated in the phase 2 evaluation, which provided manually annotated EVENT and TIMEX3 spans with their attributes. The system was trained on the "train" set and evaluated on the "test" set. An error analysis was also performed on the "development" set.

Key Insights and Ideas

  • cTAKES-Temporal as a strong baseline: The existing cTAKES-Temporal system provided a solid foundation, outperforming the median F-measure of participating systems.
  • Feature engineering challenges: Adding new features like MDP and IBS did not yield significant improvements, suggesting that the existing features were already quite comprehensive or that the new features had overlap.
  • Error analysis is key: The most significant insights came from analyzing errors. False negatives (missed relations) were a much larger problem than false positives.
  • Cross-sentence and unknown terms are major hurdles: The paper highlights cross-sentence relations and "unknown tokens" (terms not seen in training data) as primary sources of errors.
  • Clinical text complexity: The inherent complexity of clinical narratives, including negations, enumerations, and specific formatting (like newlines), poses substantial challenges.

Key Results

  • Run 1 Performance: Scored approximately 9% higher than the median F-measure and 4% short of the best system on the test set.
  • Run 2 Performance: Adding new features resulted in only a tiny increase in precision and recall (0.001), indicating limited impact.
  • Error Distribution: For false positives, a significant portion was categorized as "questionable" or "correct," suggesting the system's actual precision might be higher than reported. For false negatives, cross-sentence relations and unknown tokens each accounted for 26% of errors.
  • Out-of-Competition Experiments: Extending the search space to include pairs separated by newlines improved recall by 2% and F-measure by 1%, but this was a modest improvement given the prevalence of cross-newline errors.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Limited Error Analysis: Due to time constraints, the error analysis on the development set was based on a small sample.
  • Feature Overlap: The limited impact of new features might be due to overlap with existing ones.
  • UNK Handling: The experiments with sub-word features (character n-grams) and word embeddings for UNK tokens did not yield significant improvements, suggesting a need for more advanced techniques.
  • Cross-Newline Errors: While extending the search space helped, there's still significant room for improvement in handling relations split by newlines.
  • Future Research: Exploring alternative methods for dealing with UNKs and further refining techniques for cross-sentence and cross-newline relations are suggested.

Summary updated

BibTeX
@inproceedings{leeuwenberg2016kuleuven,
  author = {Artuur Leeuwenberg and Marie-Francine Moens},
  title = {{KULeuven-LIIR at SemEval 2016 Task 12: Detecting Narrative Containment in Clinical Records}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  url = {https://openreview.net/forum?id=_eShcCFH5-bc},
}
Figure