Translating clinical notes into quantitative measures—a real-world observation on the response to cholinesterase inhibitors or selective serotonin reuptake inhibitors prescribed to outpatients with dementia using electronic medical records

Jill Chotiyanonta, Kengo Onda, Milap A. Nowrangi, Xin Li, Xin Xu, Roy J. Adams, Constantine G. Lyketsos, Peter P. Zandi, Kenichi OishiFrontiers in Pharmacology 2023 · DOI

TL;DR

Researchers developed a simplified 3-point scoring system called NOTE (NOte-based evaluation method for Treatment Efficacy) to transform unstructured clinical text into quantitative data. By applying this to electronic medical records (EMRs), they found that real-world responder rates for dementia medications closely mirror those seen in highly controlled clinical trials, validating the use of routine clinical notes for large-scale observational research.

What Problem Does This Paper Address?

Clinical trials are the gold standard for drug efficacy, but they operate under "lab conditions"—strict inclusion criteria and standardized assessments that don't always reflect the "messy" reality of outpatient care. While Electronic Medical Records (EMRs) contain a wealth of real-world data, much of the critical information regarding how a patient actually feels or behaves is trapped in unstructured free-text clinical notes.

Standardized cognitive tests like the Mini-Mental State Examination (MMSE) are not always performed at every visit, creating data gaps. This paper addresses the challenge of "translating" the qualitative observations of clinicians and caregivers into a reliable, quantitative metric to measure how well dementia drugs (specifically cholinesterase inhibitors and SSRIs) work in a diverse, real-world population.

Methods

The researchers performed a retrospective analysis of EMR data from Johns Hopkins (2010–2021). The core innovation is the NOTE (NOte-based evaluation method for Treatment Efficacy) scale:

  • The Scale: A simplified three-point Likert scale (1 = improve, 2 = no change, 3 = worsen).
  • Classification: Notes were categorized into "cognitive" (memory, focus) and "psychiatric" (anxiety, agitation) domains.
  • Validation: To ensure NOTE was robust, they compared it against the CIBIC-plus, a more granular seven-point scale typically used in clinical trials, and against objective changes in MMSE scores.
  • Reliability: They used Krippendorff’s alpha and Intraclass Correlation Coefficients (ICC) to measure inter-rater reliability between different reviewers.

Datasets

The study utilized a curated dataset derived from the Johns Hopkins EMR system via the Richman Family Precision Medicine Center of Excellence in Alzheimer’s Disease.

  • Initial Screening: 772 patients.
  • Final Cohort: After applying exclusion criteria (e.g., missing start dates, insufficient follow-up), the study analyzed 225 SSRI cases and 115 CEI cases.
  • Diagnosis Mix: Unlike trials that often focus solely on Alzheimer’s, this dataset included Vascular Dementia (VD), mixed dementia, and other degenerative diseases.

Key Insights and Ideas

  • Simplicity is a Feature, Not a Bug: The researchers found that the simplified 3-point NOTE scale was actually more suitable for EMR analysis than the 7-point CIBIC-plus. Clinical notes often lack the specific nuances (e.g., "markedly" vs. "moderately" improved) required for more complex scales, making the 3-point version more reliable for retrospective review.
  • Real-World Validation: Despite the heterogeneity of the patients (varying ages, comorbidities, and dementia types), the responder rates were remarkably similar to those reported in controlled trials. This suggests that clinician "impressions" recorded in notes are a highly valid proxy for formal clinical trial outcomes.
  • The "Stable" Definition: In dementia care, "no change" in cognition is often considered a treatment success (a responder), whereas for psychiatric symptoms like agitation, "no change" is typically seen as a non-response. The study adjusted its definition of "responder" based on the domain being treated.

Key Results

  • Cholinesterase Inhibitors (CEIs): 61.8% of cases were classified as responders (27.0% improved, 34.8% remained stable).
  • SSRIs: 69.3% of cases showed improvement in neuropsychiatric symptoms.
  • Reliability: The NOTE scale showed near-perfect inter-rater reliability (ICC of 1.0 for cognitive notes and 0.96 for psychiatric notes).
  • Correlation: NOTE scores correlated strongly with CIBIC-plus scores (ρ = 0.93 to 0.99) and significantly with objective changes in MMSE scores, proving that the qualitative notes reflect underlying clinical reality.

Limitations and Future Work

  • The Manual Bottleneck: The primary limitation is that the EMR review was manual and labor-intensive. This limited the sample size and prevented the analysis of complex variables like drug-drug interactions or the impact of specific comorbidities.
  • Subjectivity: While reliability was high, the method still relies on human interpretation of clinical text, which can be prone to bias.
  • Future Direction: The authors explicitly state that this manually reviewed dataset is intended to serve as a "gold standard" reference set. The next step is to use this data to train Natural Language Processing (NLP) models to automate the extraction of NOTE scores, enabling the analysis of tens of thousands of records across different healthcare systems.

Summary updated

Abstract

<b>Objective:</b> Cholinesterase inhibitors (CEIs) are prescribed for dementia to maintain or improve memory. Selective serotonin reuptake inhibitors (SSRIs) are also prescribed to manage psychiatric symptoms seen in dementia. What proportion of outpatients actually responds to these drugs is still unclear. Our objective was to investigate the responder rates of these medications in an outpatient setting using the electronic medical record (EMR). <b>Methods:</b> We used the Johns Hopkins EMR system to identify patients with dementia who were prescribed a CEI or SSRI for the first time between 2010 and 2021. Treatment effects were assessed through routinely documented clinical notes and free-text entries in which healthcare providers record clinical findings and impressions of patients. Responses were scored using a three-point Likert scale named the NOte-based evaluation method for Treatment Efficacy (NOTE) in addition to the Clinician's Interview-Based Impression of Change Plus caregiver input (CIBIC-plus), a seven-point Likert scale used in clinical trials. To validate NOTE, the relationships between NOTE and CIBIC-plus and between NOTE and change in MMSE (Mini-Mental State Examination) before and after medication were examined. Inter-rater reliability was evaluated using Krippendorff's alpha. The responder rates were calculated. <b>Results:</b> NOTE showed excellent inter-rater reliability and correlated well with CIBIC-plus and changes in MMSEs. Out of 115 CEI cases, 27.0% reported improvement and 34.8% reported stable symptoms in cognition; out of 225 SSRI cases, 69.3% reported an improvement in neuropsychiatric symptoms. <b>Conclusion:</b> NOTE showed high validity in measuring the pharmacotherapy effects based on unstructured clinical entries. Although our real-world observation included various types of dementia, the results were remarkably similar to what was reported in controlled clinical trials of Alzheimer's disease and its related neuropsychiatric symptoms.
BibTeX
@inproceedings{chotiyanonta2023translating,
  author = {Jill Chotiyanonta and Kengo Onda and Milap A. Nowrangi and Xin Li and Xin Xu and Roy J. Adams and Constantine G. Lyketsos and Peter P. Zandi and Kenichi Oishi},
  title = {{Translating clinical notes into quantitative measures—a real-world observation on the response to cholinesterase inhibitors or selective serotonin reuptake inhibitors prescribed to outpatients with dementia using electronic medical records}},
  booktitle = {Frontiers in Pharmacology},
  year = {2023},
  doi = {10.3389/fphar.2023.1177026},
}
Figure