PriMock57: A Dataset Of Primary Care Mock Consultations

Alex Papadopoulos-Korfiatis, Francesco Moramarco, Radmila Sarac, Aleksandar SavkovACL 2022 · DOI

Introduction

The administrative burden on primary care clinicians is a significant contributor to burnout. While Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) offer potential solutions through automated note generation, research in this domain is hindered by a "data moat." Sensitive Personal Health Information (PHI) means that most clinical datasets are proprietary, preventing standardized benchmarking. PriMock57 addresses this by providing a high-quality, open-access dataset of 57 mock primary care consultations, including audio, manual transcripts, and clinician-authored notes.

Methodology

To create a realistic yet privacy-compliant dataset, the authors employed a pragmatic approach involving mock consultations. Seven clinicians and 57 actors participated in telemedicine-based sessions. Actors were provided with "case cards" detailing simulated medical histories and symptoms for common conditions such as asthma and migraines.

The PriMock57 data collection and processing workflow.

The workflow involved capturing separate audio channels for the clinician and patient. These recordings were then professionally transcribed at the utterance level to establish a ground truth. Finally, clinicians authored formal consultation notes following the Subjective, Objective, Assessment, Plan (SOAP) format.

Dataset Characteristics

The PriMock57 dataset comprises 57 consultations, totaling approximately 8.6 hours of audio. It includes a diverse range of clinical scenarios, from cardiovascular issues to mental health. For benchmarking, the authors evaluated several ASR engines (e.g., Kaldi, Google Cloud STT, Amazon Transcribe Medical) and summarization models (e.g., BART, BERT).

Results and Discussion

The study yielded several key insights into clinical ASR and note generation:

ASR Performance

Surprisingly, general-domain ASR engines outperformed medical-specific ones. Google Cloud STT and Azure STT achieved the lowest Word Error Rates (WER) and the highest Extracted Clinical Concepts Accuracy (ECCA).

ASR performance metrics across different models and demographic factors.

Conversation Dynamics and Summarization

Analysis showed a distinct shift in conversation dominance: patients typically lead the first half of the consultation describing symptoms, while clinicians lead the second half during diagnosis and management.

Regarding note generation, standard summarization models like BART-CNN performed poorly, often failing to beat a random sentence selection baseline. Only a version of BART fine-tuned on real clinical data (BART-finet) showed significant improvement, highlighting that clinical scribing requires structural knowledge of the SOAP format rather than simple summarization.

Conclusion

PriMock57 serves as a vital public benchmark for the clinical NLP community. While limited by its scale (57 consultations) and the inherent differences between mock and real-world interactions, it provides a necessary foundation for developing "medically aware" models. Future work will focus on scaling such datasets and improving model robustness to the linguistic complexities of authentic patient-doctor dialogue.

Summary updated

BibTeX
@inproceedings{papadopouloskorfiatis2022primock,
  author = {Alex Papadopoulos-Korfiatis and Francesco Moramarco and Radmila Sarac and Aleksandar Savkov},
  title = {{PriMock57: A Dataset Of Primary Care Mock Consultations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-short.65},
}
Figure