Introduction
The administrative burden on primary care clinicians is a significant contributor to burnout. While Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) offer potential solutions through automated note generation, research in this domain is hindered by a "data moat." Sensitive Personal Health Information (PHI) means that most clinical datasets are proprietary, preventing standardized benchmarking. PriMock57 addresses this by providing a high-quality, open-access dataset of 57 mock primary care consultations, including audio, manual transcripts, and clinician-authored notes.
Methodology
To create a realistic yet privacy-compliant dataset, the authors employed a pragmatic approach involving mock consultations. Seven clinicians and 57 actors participated in telemedicine-based sessions. Actors were provided with "case cards" detailing simulated medical histories and symptoms for common conditions such as asthma and migraines.
The workflow involved capturing separate audio channels for the clinician and patient. These recordings were then professionally transcribed at the utterance level to establish a ground truth. Finally, clinicians authored formal consultation notes following the Subjective, Objective, Assessment, Plan (SOAP) format.
Dataset Characteristics
The PriMock57 dataset comprises 57 consultations, totaling approximately 8.6 hours of audio. It includes a diverse range of clinical scenarios, from cardiovascular issues to mental health. For benchmarking, the authors evaluated several ASR engines (e.g., Kaldi, Google Cloud STT, Amazon Transcribe Medical) and summarization models (e.g., BART, BERT).
Results and Discussion
The study yielded several key insights into clinical ASR and note generation:
ASR Performance
Surprisingly, general-domain ASR engines outperformed medical-specific ones. Google Cloud STT and Azure STT achieved the lowest Word Error Rates (WER) and the highest Extracted Clinical Concepts Accuracy (ECCA).
Conversation Dynamics and Summarization
Analysis showed a distinct shift in conversation dominance: patients typically lead the first half of the consultation describing symptoms, while clinicians lead the second half during diagnosis and management.
Regarding note generation, standard summarization models like BART-CNN performed poorly, often failing to beat a random sentence selection baseline. Only a version of BART fine-tuned on real clinical data (BART-finet) showed significant improvement, highlighting that clinical scribing requires structural knowledge of the SOAP format rather than simple summarization.
Conclusion
PriMock57 serves as a vital public benchmark for the clinical NLP community. While limited by its scale (57 consultations) and the inherent differences between mock and real-world interactions, it provides a necessary foundation for developing "medically aware" models. Future work will focus on scaling such datasets and improving model robustness to the linguistic complexities of authentic patient-doctor dialogue.
Summary updated