Time Expressions in Mental Health Records for Symptom Onset Extraction

Natalia Viani, Lucia Yin, Joyce Kam, Ayunni Alawi, André Bittar, Rina Dutta, Rashmi Patel, Robert Stewart, Sumithra VelupillaiEMNLP 2018

TL;DR

This paper introduces the first annotated corpus for temporal expressions in mental health records, specifically for schizophrenia patients, to aid in calculating the duration of untreated psychosis (DUP). They adapted existing NLP tools (SUTime and HeidelTime) to this domain, achieving promising results and highlighting the unique challenges of temporal information in mental health narratives.

What Problem Does This Paper Address?

For psychiatric disorders like schizophrenia, the duration of untreated psychosis (DUP) is a critical factor influencing patient outcomes. Shorter DUPs are linked to better cognitive function and less severe symptoms. Accurately measuring DUP requires identifying when symptoms first appeared and when treatment began, which necessitates extracting temporal information from clinical records. However, mental health records are often unstructured, narrative-heavy, and contain temporal expressions that are vague or context-dependent, making them difficult for standard NLP tools to process. This paper addresses the challenge of accurately identifying and normalizing these time expressions within this specific clinical domain.

Methods

The core of this work involves creating a specialized annotated corpus and adapting existing NLP tools.

  • Corpus Creation: The researchers manually annotated a dataset of anonymized electronic health records (EHRs) from schizophrenia patients. They developed domain-specific annotation guidelines, drawing inspiration from general NLP and other clinical domains (like ICU and oncology) but adapting them to the unique characteristics of mental health narratives. A key innovation is the introduction of an "Age-related" TIMEX type to capture crucial temporal information often expressed in relation to a patient's age.
  • NLP Tool Adaptation: They took two established time expression extraction systems, SUTime and HeidelTime, and adapted them to the mental health domain. This involved an iterative process of running the original systems, analyzing errors on a development set, and then modifying the systems' rules to improve performance. The focus was on improving recall for partial matches, acknowledging the difficulty in exact span identification.

Datasets

  • Anonymized Mental Health Records: The study utilized anonymized EHRs from the Clinical Record Interactive Search (CRIS) database, specifically from patients diagnosed with schizophrenia. The data was curated to include documents within the first three months of patient presentation and the longest document per patient to maximize relevant information.
  • Gold Standard Corpus: A subset of 52 documents was manually annotated to create a gold standard corpus for time expression extraction. This corpus is unique in its focus on the mental health domain and its inclusion of age-related temporal expressions.

Key Insights and Ideas

  • Mental Health Narratives are Unique: Unlike more structured clinical domains (ICU, oncology), mental health records are characterized by extensive free text, scattered information, and less reliance on structured fields. This makes temporal information harder to pinpoint.
  • "Age-related" Expressions are Crucial: The introduction of the "Age-related" TIMEX type is a significant contribution. These expressions (e.g., "at the age of 18," "when he was a child") are vital for determining symptom onset in mental health, as patients or relatives often report these events vaguely.
  • Vagueness is Common: Temporal expressions in this domain are often vague (e.g., "this morning," "at night") or relative to the patient's age, posing challenges for normalization.
  • Domain-Specific Rules are Essential: General-purpose NLP tools struggle with the nuances of mental health records. Adapting these tools requires specific rule modifications to handle domain-specific temporal expressions (e.g., drug prescription frequencies like "OD," or specific time intervals like "15 minute" used for patient observation).
  • Partial vs. Strict Matching Matters: The difference between partial and strict IAA measures highlights the difficulty in precisely identifying the boundaries (spans) of time expressions, a common challenge in NLP.

Key Results

  • Corpus Statistics: The gold standard corpus contains 3,413 time expressions, averaging 65.6 per document. Dates are the most prevalent type (55.8%), followed by Durations (16.5%), Times (10.7%), Frequencies (8.1%), and the newly introduced Age-related type (8.9%).
  • Inter-Annotator Agreement (IAA): For partial matches, IAA was 78% (precision/recall) and 77% (F1 score). For strict matches, IAA was 60% for both metrics. While lower than some other clinical corpora, this is considered promising given the domain's complexity.
  • NLP System Performance:
    • Original SUTime: 72.5% F1 score on the development set.
  • Original HeidelTime: 63.6% F1 score on the development set.
  • Adapted SUTime: Achieved a final F1 score of 79.5% on the validation set.
  • Adapted HeidelTime: Achieved a final F1 score of 75.8% on the validation set.
  • The adapted SUTime achieved a higher F1 score than the human annotators' strict IAA, particularly in recall.

Limitations and Future Work

  • Normalization Not Addressed: This study focused solely on extracting time expression spans and types. The crucial step of normalizing these expressions (e.g., converting "for three years" to P3Y) was not covered but is a planned future work.
  • Contextual Disambiguation: The current adaptation did not deeply address contextual disambiguation, where the meaning of a time expression can change based on its surrounding text (e.g., "at night" as a specific time vs. a frequency). Future work will explore using word embeddings for contextual features.
  • Limited Scope: The study focused on schizophrenia patients. Extending the adaptability study to other clinical domains and corpora is planned.
  • Public Availability: Due to governance regulations, the annotated corpus cannot be made publicly available, though access procedures are in place.
  • Next Steps: The authors plan to extend annotations to include normalized values and then develop rules for extraction and normalization. They will also focus on linking clinical entities (like symptoms) to these temporal expressions to ultimately calculate DUP.

Summary updated

BibTeX
@inproceedings{viani2018time,
  author = {Natalia Viani and Lucia Yin and Joyce Kam and Ayunni Alawi and André Bittar and Rina Dutta and Rashmi Patel and Robert Stewart and Sumithra Velupillai},
  title = {{Time Expressions in Mental Health Records for Symptom Onset Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure