PhenoPad: Building AI enabled note-taking interfaces for patient encounters

Jixuan Wang, Jingbo Yang, Haochi Zhang, Helen Lu, Marta Skreta, Mia Husic, Aryan Arbabi, Nicole Sultanum, Michael BrudnoNpj Digit. Medicine 2022 · OpenReview

Abstract

Current clinical note-taking approaches cannot capture the entirety of information available from patient encounters and detract from patient-clinician interactions. By surveying healthcare providers’ current note-taking practices and attitudes toward new clinical technologies, we developed a patient-centered paradigm for clinical note-taking that makes use of hybrid tablet/keyboard devices and artificial intelligence (AI) technologies. PhenoPad is an intelligent clinical note-taking interface that captures free-form notes and standard phenotypic information via a variety of modalities, including speech and natural language processing techniques, handwriting recognition, and more. The output is unobtrusively presented on mobile devices to clinicians for real-time validation and can be automatically transformed into digital formats that would be compatible with integration into electronic health record systems. Semi-structured interviews and trials in clinical settings rendered positive feedback from both clinicians and patients, demonstrating that AI-enabled clinical note-taking under our design improves ease and breadth of information captured during clinical visits without compromising patient-clinician interactions. We open source a proof-of-concept implementation that can lay the foundation for broader clinical use cases.

Introduction

The "electronic health record (EHR) tax" is a well-documented burden in modern medicine, with clinicians often spending two hours on documentation for every one hour of direct patient care. This leads to two major issues: the "screen barrier," where typing reduces eye contact and impairs the clinician-patient relationship, and information loss, where the natural flow of paper notes is lost during tedious manual digitization. PhenoPad addresses these challenges by reimagining the clinical encounter through an AI-assisted, multimodal note-taking interface that combines the natural feel of handwriting with the power of automated processing.

System Design and Methodology

PhenoPad was developed through a user-centered design process, starting with surveys of 66 healthcare providers to identify workflow pain points. The system runs on a hybrid tablet/laptop (Microsoft Surface Book) paired with a custom audio capture setup consisting of a Raspberry Pi and a microphone array.

The PhenoPad system architecture and study timeline, illustrating the integration of multimodal inputs and AI processing.

The technical architecture integrates several key AI components:

  • Multimodal Input: Supports handwriting recognition (HWR), real-time speech-to-text, free-form drawings, and photo/video capture.
  • Real-time NLP: Uses a Neural Concept Recognizer (NCR) to extract clinical terms and map them to the Human Phenotype Ontology (HPO).
  • Speaker Diarization: Employs deep learning models to distinguish between clinician and patient voices in real-time.
  • Automated Disambiguation: Specialized models expand medical abbreviations and correct handwriting recognition errors.
  • Clinical Decision Support: Extracted phenotypes are used to suggest potential diagnoses via OMIM or prompt follow-up questions in real-time.

The system's speech recognition engine was fine-tuned on a language model trained on 2 million PubMed abstracts, while the diarization model utilized the VoxCeleb2 dataset.

User Interface and Key Features

PhenoPad prioritizes a "human-in-the-loop" approach, presenting AI-extracted data to the clinician for real-time validation rather than attempting full automation. This ensures accuracy while the encounter details are still fresh in the clinician's mind.

A key innovation is the "sticker" paradigm for annotating records. Clinicians can underline or circle text in a patient's history, triggering a digital "sticky note" for new observations that remain anchored to the original record. This hybrid utility allows for the recording of physical observations that may not be spoken aloud—a critical layer of data often missed by purely audio-based digital scribes.

Evaluation and Results

The system was field-tested during 25 clinical sessions at the Hospital for Sick Children (SickKids) in Toronto. Results indicated high patient acceptance, with 92% of patients feeling comfortable with the recording and noting that the tablet allowed for better eye contact compared to standard laptop use.

Technically, the phenotype recognizer achieved a precision of 82.91%, though recall was lower at 51.08% due to the complexity of noisy clinical environments. While the Word Error Rate (WER) for speech recognition was 53.31%, it proved sufficient for the NLP layer to extract key clinical concepts effectively. Clinicians generally found the tool supportive and unobtrusive, noting it helped capture a greater breadth of information without compromising the patient interaction.

Discussion and Future Work

While PhenoPad improves documentation quality, some physicians noted that for simple visits, direct EHR entry remains faster. Future work aims to bridge this speed gap by using NLP to automatically populate note templates. Additionally, improving ASR accuracy through larger, in-domain clinical datasets and achieving seamless, direct API integration with EHR systems are primary goals. While the current iteration was heavily influenced by a Genetics department, expanding the system's decision support to other specialties like Cardiology or Orthopedics is a necessary next step for broader clinical adoption.

Summary updated

Abstract

Current clinical note-taking approaches cannot capture the entirety of information available from patient encounters and detract from patient-clinician interactions. By surveying healthcare providers’ current note-taking practices and attitudes toward new clinical technologies, we developed a patient-centered paradigm for clinical note-taking that makes use of hybrid tablet/keyboard devices and artificial intelligence (AI) technologies. PhenoPad is an intelligent clinical note-taking interface that captures free-form notes and standard phenotypic information via a variety of modalities, including speech and natural language processing techniques, handwriting recognition, and more. The output is unobtrusively presented on mobile devices to clinicians for real-time validation and can be automatically transformed into digital formats that would be compatible with integration into electronic health record systems. Semi-structured interviews and trials in clinical settings rendered positive feedback from both clinicians and patients, demonstrating that AI-enabled clinical note-taking under our design improves ease and breadth of information captured during clinical visits without compromising patient-clinician interactions. We open source a proof-of-concept implementation that can lay the foundation for broader clinical use cases.
BibTeX
@inproceedings{wang2022phenopad,
  author = {Jixuan Wang and Jingbo Yang and Haochi Zhang and Helen Lu and Marta Skreta and Mia Husic and Aryan Arbabi and Nicole Sultanum and Michael Brudno},
  title = {{PhenoPad: Building AI enabled note-taking interfaces for patient encounters}},
  booktitle = {Npj Digit. Medicine},
  year = {2022},
  url = {https://openreview.net/forum?id=NO0F4cxTDuZ},
}
Figure