Leveraging medical context to recommend semantically similar terms for chart reviews

Cheng Ye, Bradley A. Malin, Daniel Fabbri

TL;DR

This paper introduces a "medical-context vector space" that dynamically recommends semantically similar terms for clinical chart reviews, outperforming traditional word embeddings. By considering non-textual contextual information, this approach significantly reduces the effort needed to train models for identifying relevant terms, making clinical information retrieval more efficient.

What Problem Does This Paper Address?

Clinical chart reviews, a crucial but time-consuming task for healthcare professionals, involve sifting through vast amounts of electronic medical records (EMRs) to find specific information. A common approach to speed up these reviews is "query expansion," where a search term is augmented with semantically similar terms. However, existing methods often rely on static sets of similar terms, which are insufficient because the relevance of terms can vary greatly depending on the specific chart review task (e.g., identifying a cancer stage versus finding medication information for epilepsy). Furthermore, standard natural language processing models like BERT, while powerful, don't fully capture the nuanced, non-textual context present in medical records, such as the author's department, the patient's demographics, or the structure of the note itself. This paper aims to bridge this gap by developing a method that leverages these contextual factors to dynamically generate task-appropriate similar terms.

Methods

The core innovation is the medical-context vector space. This approach represents each word not just by its textual neighbors but also by its usage across various medical contexts. These contexts are derived from EMR data and include:

  • Hospital Organizational Structure: Author's department and staff title.
  • Medical Events: Documented diagnoses (ICD codes), procedures (CPT codes), and chief complaints.
  • Demographics: Patient gender and age ranges.
  • Medical Note Structure: Note types and sections within notes.

The process involves:

  1. Preprocessing: Extracting attribute values from EMR notes and filtering stop words.
  2. Initialization: Creating a multi-dimensional vector for each word, where each dimension corresponds to a medical context, initially set to zero.
  3. Accumulation: Incrementing the vector's dimensions based on the word's occurrence within specific contexts in the notes.
  4. Normalization: Converting these counts into proportions (0.0 to 1.0) to create the final medical-context vector for each term.

The similarity between two terms is then calculated using the cosine similarity of their medical-context vectors. This results in a "medical-context similarity vector" that captures the term's relationship across all defined contexts. This vector space is then used to train supervised machine learning models to predict which terms reviewers would highlight as important for a given task.

Datasets

The study utilized de-identified electronic medical records from Vanderbilt University Medical Center (VUMC) generated between January 1, 2016, and January 2, 2017.

Three specific evaluation datasets were created, each corresponding to a chart review task:

  1. Acute Myocardial Infarction (AMI) Project: Highlighting notes related to AMI diagnoses, medications, procedures, or symptoms.
  2. Crohn's Anti-TNF Responsiveness Project: Highlighting text indicating a patient's response to anti-TNF medication for Crohn's disease.
  3. Pediatric Diabetes Note Barriers Project: Highlighting text related to barriers in documenting diabetes care plans.

These datasets were deployed on the Pybossa crowdsourcing platform and reviewed by medical researchers.

Key Insights and Ideas

  • Context is King (Even Non-Textual Context): The study powerfully demonstrates that factors beyond the immediate text (like author department or patient age) are crucial for understanding term relevance in a clinical setting.
  • Dynamic Similarity is Essential: A one-size-fits-all approach to similar terms fails. The medical-context vector space allows for task-specific term recommendations, adapting to the nuances of different chart review goals.
  • Efficiency Gains: By better identifying relevant terms, the medical-context vector space significantly reduces the amount of labeled data needed to train predictive models, a major bottleneck in medical NLP.
  • Interpretability Matters: The study explored the interpretability of the model, showing how specific medical contexts (like "Chief Complaint") have a strong influence on reviewer preferences, offering insights into why certain terms are deemed relevant.

Key Results

  • Superior Performance: The medical-context vector space consistently outperformed baseline word2vec embeddings across all three chart review tasks, achieving an average AUROC of 0.80 compared to 0.66 for word2vec.
  • Reduced Labeling Effort: The medical-context vector space significantly reduced the number of labels required to train predictive models, often by an order of magnitude (e.g., from thousands to tens of labels).
  • Interpretable Weights: Logistic regression models trained using the medical-context vectors provided interpretable weights for each medical context, indicating their relative importance in predicting reviewer preferences. For instance, the "Chief Complaint" context showed a significant positive impact across tasks.

Limitations and Future Work

The authors acknowledge several limitations and suggest avenues for future research:

  • Time Range for Medical Events: The study limited medical event contexts to a 48-hour window. Investigating different time ranges could reveal further insights.
  • Number of Medical Contexts: While ten contexts were used, exploring additional medical contexts (e.g., laboratory results) could enhance the model.
  • Reviewer Surveys: The study indicated the importance of certain contexts but suggests surveying reviewers to understand why they prefer specific terms.
  • Domain Knowledge Integration: Future work could incorporate more explicit domain knowledge into the vector space.
  • Clinical Researcher Input: Extending the system to allow clinical researchers to actively identify important medical contexts could further refine its adaptability.
  • Real-World Deployment: The authors propose testing and further developing the vector space method within existing EMR search engines.

Summary updated

Abstract

Information retrieval (IR) help clinicians answer questions posed to large collections of electronic medical records (EMRs), such as how best to identify a patient’s cancer stage. One of the more promising approaches to IR for EMRs is to expand a keyword query with similar terms (e.g., augmenting cancer with mets). However, there is a large range of clinical chart review tasks, such that fixed sets of similar terms is insufficient. Current language models, such as Bidirectional Encoder Representations from Transformers (BERT) embeddings, do not capture the full non-textual context of a task. In this study, we present new methods that provide similar terms dynamically by adjusting with the context of the chart review task. We introduce a vector space for medical-context in which each word is represented by a vector that captures the word’s usage in different medical contexts (e.g., how frequently cancer is used when ordering a prescription versus describing family history) beyond the context learned from the surrounding text. These vectors are transformed into a vector space for customizing the set of similar terms selected for different chart review tasks. We evaluate the vector space model with multiple chart review tasks, in which supervised machine learning models learn to predict the preferred terms of clinically knowledgeable reviewers. To quantify the usefulness of the predicted similar terms to a baseline of standard word2vec embeddings, we measure (1) the prediction performance of the medical-context vector space model using the area under the receiver operating characteristic curve (AUROC) and (2) the labeling effort required to train the models. The vector space outperformed the baseline word2vec embeddings in all three chart review tasks with an average AUROC of 0.80 versus 0.66, respectively. Additionally, the medical-context vector space significantly reduced the number of labels required to learn and predict the preferred similar terms of reviewers. Specifically, the labeling effort was reduced to 10% of the entire dataset in all three tasks. The set of preferred similar terms that are relevant to a chart review task can be learned by leveraging the medical context of the task.
BibTeX
@article{ye2021leveraging,
  author = {Cheng Ye and Bradley A. Malin and Daniel Fabbri},
  title = {{Leveraging medical context to recommend semantically similar terms for chart reviews}},
  journal = {BMC Medical Informatics Decis. Mak.},
  year = {2021},
  doi = {10.1186/s12911-021-01724-2},
}
Figure