Listwise temporal ordering of events in clinical notes

Serena Jeblee, Graeme HirstEMNLP 2018

TL;DR

This paper introduces a novel "listwise" approach to ordering events in clinical notes, moving beyond traditional pairwise comparisons to create a more consistent and comprehensive timeline. By framing temporal ordering as a ranking problem, they develop new evaluation metrics and a baseline model that outperforms random ordering and simple mention order.

What Problem Does This Paper Address?

Understanding the temporal sequence of events in clinical notes is crucial for many downstream medical NLP tasks, such as predicting patient readmissions or summarizing medical histories. However, clinical notes often present events out of chronological order. Traditional methods tackle this by classifying temporal relationships between pairs of events. This pairwise approach suffers from several drawbacks: it generates a massive number of pairs (n²), many of which have no defined relationship, leading to imbalanced classification problems. Furthermore, pairwise methods can produce inconsistent relationships (e.g., A before B, B before C, but C before A), and are often limited to events mentioned within the same sentence or a small window, missing important long-distance temporal connections. This paper aims to address these limitations by proposing a listwise ordering approach that inherently captures all pairwise relationships and provides a more robust and consistent temporal timeline.

Methods

The core contribution of this paper is the shift from pairwise event relation classification to a listwise temporal ordering approach. Instead of predicting the relationship between every pair of events, they aim to generate a single, ordered list of all medically relevant events for a patient. This listwise representation is framed as a ranking task.

They adapt the ListNet algorithm, a listwise ranking method commonly used in information retrieval, for this task. The ListNet model learns a ranking function that assigns a relevance score to each event, and the events are then ordered based on these scores.

To handle the nuances of temporal relations, they incorporate event time information (start, end, and overlap times) when available. For the THYME dataset, they map various temporal relations (like BEFORE, AFTER, OVERLAP, CONTAINS) into a simplified set suitable for their listwise ordering by start time.

Crucially, the paper also introduces new listwise evaluation metrics to better assess the quality of the generated event timelines. These include:

  • Mean Squared Error (MSE): Measures the absolute difference between the predicted and correct rank for each event.
  • Pairwise Ordering Accuracy (POA): Accounts for slight deviations in rank, considering events to be correctly ranked if their predicted ranks are within a small epsilon (ε) of each other. This metric is more forgiving than strict rank matching.
  • Gold-standard Pairwise Relation Recall (GPR): Extracts pairwise relations from the generated list and compares them to the gold-standard annotations, reporting recall for different relation types.

Datasets

The primary dataset used in this work is the THYME corpus. This corpus contains de-identified clinical notes with human-annotated times, events, and temporal relations, following the TimeML schema. The authors utilize the provided train/dev/test split and the gold-standard annotations, including document creation time (DCT) relations. They note that a limitation of the THYME dataset is that event annotations are often applied to single words, even when a phrase would be more appropriate.

Key Insights and Ideas

  • Listwise ordering inherently resolves pairwise inconsistencies: By generating a single ordered list, the model avoids the cyclical and contradictory relationships that can arise from pairwise classification.
  • Ranking as a solution for temporal ordering: Framing temporal ordering as a ranking problem allows the application of established ranking algorithms like ListNet, opening up new avenues for research.
  • New metrics are essential for listwise evaluation: Traditional ranking metrics like NDCG and MAP are not ideal for temporal ordering where the order of all events matters. The proposed MSE and POA metrics provide a more suitable evaluation framework.
  • Event duration and overlap can be implicitly captured: While the primary ordering is by start time, the inclusion of time interval information allows for future exploration of ordering based on end times or overlap.

Key Results

The paper demonstrates that their ListNet-based ranking model outperforms random ordering and ordering by mention in the text on the THYME test set, as measured by MSE and POA. For instance, the ListNet model achieved an MSE of 0.072 and a POA of 0.517, significantly better than random ordering (MSE 0.170, POA 0.366) and text order (MSE 0.148, POA 0.413).

When comparing their listwise approach to a pairwise classification model (a feed-forward neural network), they found that the pairwise model results often contained cycles, making it impossible to extract a consistent listwise ordering. This highlights a key advantage of their proposed method.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Simplified relation set: The current listwise ordering method primarily handles simple relations (BEFORE, AFTER, OVERLAP) and maps more complex ones. Future work could incorporate a richer set of temporal relations.
  • Ordering by start time: The current model orders events by their start times. The authors suggest exploring ordering by end time or by considering event overlaps.
  • Handling relative time phrases: The current approach focuses on absolute or interval time information. Future work could better handle relative time phrases (e.g., "two days later").
  • Event representation: The paper notes the limitation of single-word event annotations in the THYME dataset and suggests that more phrase-based annotations would be beneficial.
  • Improving the ranking model: While ListNet shows promise, there is still room for improvement in its ability to capture relative ordering.

The authors also mention that their list conversion code is publicly available, encouraging further research in this area.

Summary updated

BibTeX
@inproceedings{jeblee2018listwise,
  author = {Serena Jeblee and Graeme Hirst},
  title = {{Listwise temporal ordering of events in clinical notes}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure