Automatic Patient Note Assessment without Strong Supervision

Jianing Zhou, Vyom Nayan Thakkar, Rachel Yudkowsky, Suma Bhat, William F. BondLOUHI @ EMNLP 2022 · DOI

TL;DR

This paper tackles the challenge of automatically assessing patient notes, a crucial but labor-intensive task in medical education. They introduce a novel weakly supervised NLP framework that leverages multi-level transfer learning to achieve state-of-the-art performance, even with limited labeled data, by effectively combining natural language inference and named entity recognition tasks.

What Problem Does This Paper Address?

Medical training heavily relies on physicians accurately documenting patient encounters in written notes. Assessing these notes is vital for feedback and skill development, but it's a time-consuming and expensive process for human experts. The USMLE Step 2 Clinical Skills exam, for instance, generated over 330,000 patient notes annually, requiring extensive manual grading. Traditional supervised NLP methods struggle here due to the low-resource scenario: creating large, labeled datasets is impractical, especially as test cases evolve. Furthermore, human grading is prone to bias and requires significant training. This paper aims to automate patient note assessment, specifically by judging if a given checklist item is present in the note and identifying the corresponding phrases, without relying on extensive manual annotations.

Methods

The authors propose a two-pronged approach:

  • Unsupervised Baseline: A pipeline framework using Amazon Comprehend Medical for medical entity extraction. This serves as a basic starting point, demonstrating the challenges of a zero-resource scenario.
  • Weakly Supervised Method: This is the core contribution. It's a multi-task learning framework that leverages multi-level transfer learning:
  • Data-Level Transfer: The model can learn from "out-of-domain" data (notes from different clinical cases) to improve performance on new, unseen cases.
  • Task-Level Transfer: The model jointly trains two related tasks:
  • Natural Language Inference (NLI): Determining if a checklist item is "entailed" by the patient note.
  • Named Entity Recognition (NER): Identifying the specific phrases in the patient note that correspond to the checklist item.
  • Mutual Feedback Mechanism: The outputs of the NLI and NER tasks are used to enhance the input for each other, creating a synergistic learning process.
  • BERT Encoder: A pre-trained BERT model is used as the shared encoder for extracting contextual embeddings from both patient notes and checklist items.

Datasets

The study utilizes two main datasets:

  • Self-Collected Datasets: These are case-specific datasets focusing on two clinical scenarios: headache and abdominal pain. Data was collected from 30 examinees, with 25 used for fine-tuning and 5 for testing. The notes were annotated by physician faculty, achieving high inter-annotator agreement (above 0.9).
  • USMLE® Step 2 Clinical Skills Patient Note Dataset: This is a larger, publicly available dataset containing over 43,000 patient note history portions from 10 clinical cases. For their study, they used a subset of 100 annotated patient notes per case, referred to as the "generic dataset," which was not large enough for direct training but served as a diverse, related dataset for transfer learning.

Key Insights and Ideas

  • The Power of Multi-Task Learning: Jointly training NLI and NER tasks significantly boosts performance. Identifying relevant phrases (NER) directly aids in determining entailment (NLI), and vice-versa, creating a virtuous cycle.
  • Transfer Learning is Key for Low-Resource Scenarios: The proposed weakly supervised method demonstrates strong data-level transfer learning. Even training on generic data (out-of-domain) and then fine-tuning on a small amount of in-domain data leads to substantial performance gains, highlighting its practicality for new clinical cases.
  • Small Amounts of In-Domain Data Go a Long Way: The study shows that only a small number of in-domain examples are needed for fine-tuning to achieve satisfactory performance, reinforcing the method's efficiency.
  • Weak Supervision Outperforms Unsupervised: The weakly supervised approach significantly outperforms the unsupervised baseline, underscoring the benefit of even limited labels and task relationships.

Key Results

  • NLI Task: The weakly supervised method achieved an average accuracy of 0.93 on the headache dataset and 0.91 on the abdominal pain dataset, significantly outperforming the NLI baseline (around 0.87-0.88) and the unsupervised method (around 0.63-0.73).
  • NER Task: The weakly supervised method achieved an F1 score of 0.61 on the headache dataset and 0.62 on the abdominal pain dataset, again outperforming the NER baseline (around 0.53-0.55).
  • State-of-the-Art Performance: The paper claims their weakly supervised method achieves SOTA performance on both NLI and NER tasks for automatic patient note assessment.
  • Data-Level Transfer: Even when tested on out-of-domain data without fine-tuning, the model performed well, and fine-tuning on a small amount of in-domain data further improved results. For example, performance improved from 0.87 to 0.89 when fine-tuned on abdominal pain and tested on headache.

Limitations and Future Work

  • NER Performance Still Limited: While improved, the performance on the NER task (identifying relevant phrases) is still considered limited compared to the NLI task. The authors suggest future work should focus on developing more effective transfer learning mechanisms for phrase identification.
  • Reliance on In-Domain Data: The weakly supervised model still requires some in-domain data for fine-tuning to achieve optimal performance, as the unsupervised model's performance is unsatisfactory.
  • Small Test Set Size: The study used patient notes from only 5 examinees for testing, which is noted as a limitation. Future work should validate these results with larger samples and a wider variety of cases.
  • Practical Impact: The system is undergoing pilot testing with learners and faculty to assess its impact on providing immediate and automated feedback, which could make debriefing more focused and efficient.

Summary updated

BibTeX
@inproceedings{zhou2022automatic,
  author = {Jianing Zhou and Vyom Nayan Thakkar and Rachel Yudkowsky and Suma Bhat and William F. Bond},
  title = {{Automatic Patient Note Assessment without Strong Supervision}},
  booktitle = {LOUHI @ EMNLP},
  year = {2022},
  doi = {10.18653/v1/2022.louhi-1.14},
}
Figure