Leveraging Natural Language Processing and Clinical Notes for Dementia Detection

Ming Liu, Richard Beare, Taya Collyer, Nadine Andrew, Velandai SrikanthACL 2023

TL;DR

This paper explores using deep learning models, specifically a fine-tuned ClinicalBioBERT, to detect dementia from in-hospital clinical notes. While the model shows strong performance in a binary classification (dementia vs. non-dementia), it struggles with a more nuanced three-class classification, suggesting that current NLP models may need to incorporate more domain-specific knowledge.

What Problem Does This Paper Address?

Dementia is a growing global health concern, and early detection is crucial for better management. While neuroimaging and spontaneous speech analysis have seen progress, leveraging readily available clinical notes for dementia detection remains an underexplored area. Clinical notes, however, present unique challenges: they are often lengthy, contain abbreviations and spelling errors, and require understanding long-range dependencies. This paper aims to bridge this gap by investigating the efficacy of advanced NLP techniques on this rich but complex data source.

Methods

The core of this research involves fine-tuning a pre-trained language model, ClinicalBioBERT, on a dataset of in-hospital clinical notes. ClinicalBioBERT is chosen for its domain-specific pre-training on medical text. To handle the input length limitations of BERT-based models (typically 512 tokens), the authors employ a strategy of filtering and aggregating notes to create a patient-level summary. This involves keeping notes with at least 10 annotated UMLS concepts and using the latest notes to represent the patient. The fine-tuned model is then evaluated on both binary (dementia vs. non-dementia) and three-class (dementia vs. non-dementia vs. uncertain) classification tasks. A keyword-based approach using UMLS concepts is also used as a baseline.

Datasets

The study utilizes a newly collected dataset of 954 in-hospital clinical notes from a local hospital in Melbourne. Patients are labeled with dementia/non-dementia/uncertainty based on clinical assessment and telephone interviews. The dataset is divided into four cohorts:

  • Cohort 1a: Diagnosed with dementia (CDAMS, Positive)
  • Cohort 1b: No final diagnosis (CDAMS, Uncertain)
  • Cohort 2a: Diagnosed as non-dementia (TICS-M, Negative)
  • Cohort 2b: No diagnosis (TICS-M, Uncertain)

This dataset is a novel contribution, providing labeled clinical notes for dementia detection research.

Key Insights and Ideas

  • Clinical notes are a valuable, yet challenging, data source: The paper highlights the potential of clinical notes for dementia detection, but also underscores the difficulties posed by their length, informality, and complexity.
  • Binary classification is achievable, but nuance is hard: The fine-tuned ClinicalBioBERT performs well in distinguishing between dementia and non-dementia, but struggles significantly when introducing an "uncertain" category. This suggests that current models might be good at identifying clear cases but less adept at handling ambiguity.
  • Structured knowledge (UMLS) might be more powerful than raw embeddings: The analysis shows that representations derived from UMLS concepts are more effective at distinguishing between dementia and non-dementia patients than the [CLS] token embeddings from ClinicalBioBERT. This points to the importance of incorporating explicit medical knowledge into NLP models.
  • The "low annotation problem" persists: Even with a dataset of 954 patients, the authors note that annotation in the medical setting is expensive and time-consuming, limiting the granularity of labels (patient-level vs. document-level).

Key Results

The fine-tuned ClinicalBioBERT model achieved an accuracy of 0.814 in the binary classification setting (dementia vs. non-dementia). However, its performance dropped significantly in the three-class setting, with an accuracy of 0.381. This represents a decrease of approximately 20% from the binary to the three-class setting for both the BERT model and the keyword-based baseline. The UMLS concept representation, visualized using t-SNE, showed better separation of positive and negative dementia patients compared to the ClinicalBioBERT embeddings.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited sample size and diversity: The dataset, while novel, is from a single hospital and may not fully represent the broader patient population.
  • Patient-level labels only: Due to annotation costs, document-level labels were not obtained, which could provide richer insights.
  • Lack of cross-label checks: The "uncertain" category might contain patients with undiagnosed dementia.

Future work suggested includes:

  • Leveraging more human prior knowledge and experimenting with both statistical and deep neural models.
  • Exploring more structured patient representations using knowledge graphs.
  • Expanding the dataset with notes from diverse hospitals and age groups.
  • Developing more model interpretability and human error analysis.
  • Considering traditional statistical models like Logistic Regression with biomedical concept features.

Summary updated

BibTeX
@inproceedings{liu2023leveraging,
  author = {Ming Liu and Richard Beare and Taya Collyer and Nadine Andrew and Velandai Srikanth},
  title = {{Leveraging Natural Language Processing and Clinical Notes for Dementia Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure