PoCaP Corpus: A Multimodal Dataset for Smart Operating Room Speech Assistant using Interventional Radiology Workflow Analysis

Kubilay Can Demir, Matthias May, Axel Schmid, Michael Uder, Katharina Breininger, Tobias Weise, Andreas Maier, Seung Hee YangarXiv 2022 · arXiv

TL;DR

The PoCaP Corpus is a novel multimodal dataset designed to pave the way for voice-controlled smart assistants in the operating room. By combining German audio recordings, X-ray images, and device system logs from 31 real-world surgeries, the dataset enables researchers to analyze surgical workflows and train robust speech recognition models tailored to the medical domain.

What Problem Does This Paper Address?

Modern operating rooms (ORs) are highly digitalized, packed with advanced medical devices that generate massive amounts of data. While this technology improves patient safety, it also increases cognitive load for surgeons, who must split their attention between the patient and various equipment interfaces.

To alleviate this, researchers want to build intelligent workflow assistance systems—like a hands-free, voice-controlled smart assistant that can adjust an operating table or move a C-arm X-ray machine on command. However, developing these systems requires high-quality data. While plenty of surgical datasets exist, they consist almost exclusively of endoscopic or microscopic video. Modalities like speech, which are universal to almost every surgery, remain severely under-investigated due to the difficulty of recording in high-stress, sterile environments and strict data privacy laws.

Methods

The authors developed a synchronized, multi-channel data collection pipeline during 31 Port-Catheter Placement (PoCaP) interventions.

Fig. 1. The data recording setup of our corpus. During an intervention, X-ray images are recorded with a C-Arm machine and displayed on a screen in the OR. X-ray images were captured from this display. The surgeon and the assistant were equipped with personal microphones to record their speech signals. System commands were recorded from a local network utilizing a monitoring software. An RGB camera with an embedded microphone recorded the interventions. Audio data from this camera were added to our corpus and recorded videos are used solely for aiding with the annotation process.

The setup captured three distinct data streams:

  1. Audio/Speech: The primary surgeon and assistant wore high-quality wireless headsets to capture clear, two-channel verbal communication. A third ambient audio channel was captured via a GoPro camera.
  2. X-Ray Images: Instead of pulling raw medical files post-op, the team used Open Broadcaster Software (OBS) to duplicate and record the live widescreen display feed in the OR, saving frames at 1 frame per second (fps).
  3. System Commands: A software called ACSOS monitored the local network, generating a "ticket" (log file) every time a C-Arm machine function was used, tracking up to 393 different variables like physical rotation angle and pulse rate.

Because these devices were started manually, the authors used the autocorrelation of the audio channels to precisely align the ambient video with the headset audio, and used visual cues (like the launch screen of the logging software on the recorded display) to sync the system commands. Voice Activity Detection (VAD) and pre-trained Automatic Speech Recognition (ASR) models were then applied to transcribe the conversations.

Datasets

The paper introduces the PoCaP (Port Catheter Placement) Corpus. It contains data from 31 interventions performed by six different surgeons, with an average operation duration of roughly 81 minutes. The dataset is hierarchically annotated into 8 surgical phases and 31 distinct surgical steps.

Unlike well-known open datasets in the field—such as Cholec80 or HeiChole (which focus on laparoscopic gallbladder removals via endoscopic video) or CATARACTS (microscopic eye surgery video)—the PoCaP Corpus introduces speech and device logs. Due to strict institutional review board (IRB) and patient privacy constraints, this dataset cannot be made publicly available, but it serves as a blueprint for multimodal medical data collection.

Key Insights and Ideas

  • X-Rays as Workflow Landmarks: In interventional radiology, X-rays are used sparingly to minimize radiation doses to the patient. Because they are only taken during specific, critical moments (e.g., verifying guide wire or catheter positioning), the appearance of an X-ray image acts as a highly reliable "landmark" for identifying the current phase of a surgery.
  • The Challenge of "OR Noise": Not all speech in an OR is a command. Surgeons chat with assistants, discuss unrelated medical topics, or use highly localized dialects. A smart assistant must not only transcribe words accurately but also semantically filter out non-intervention-related conversations.
  • Speech is Highly Scalable: Unlike video data, which changes drastically depending on whether a surgery is open, laparoscopic, or microscopic, speech is a constant across all surgical domains, making it an ideal modality for generalized OR assistance.

Key Results

To find the best baseline for transcribing medical German in a noisy environment, the authors benchmarked six pre-trained ASR models on a manually transcribed subset of their data.

While standard models struggled heavily—one common model hit a disastrous 85.80% Word Error Rate (WER)—the stt_de_conformer_transducer_large model performed exceptionally well, achieving a 11.52% WER. This low error rate demonstrates that pre-trained, non-medical models can still be highly robust against medical terminology and ambient OR noise, proving the feasibility of speech-controlled systems.

Limitations and Future Work

The primary limitation of the PoCaP Corpus is its scale and localization: it features 31 interventions from a single medical center (University Hospital Erlangen) involving just six surgeons.

Moving forward, the authors plan to expand the dataset to include more surgeries and diverse environments. Future algorithmic work will focus on translating medical dialects into standard German, mapping synonyms (different phrases used by surgeons to mean the same command), and training machine learning models to automatically predict the remaining time in a surgery based on the live multimodal stream.

Summary updated

Abstract

This paper presents a new multimodal interventional radiology dataset, called PoCaP (Port Catheter Placement) Corpus. This corpus consists of speech and audio signals in German, X-ray images, and system commands collected from 31 PoCaP interventions by six surgeons with average duration of 81.4 $\pm$ 41.0 minutes. The corpus aims to provide a resource for developing a smart speech assistant in operating rooms. In particular, it may be used to develop a speech controlled system that enables surgeons to control the operation parameters such as C-arm movements and table positions. In order to record the dataset, we acquired consent by the institutional review board and workers council in the University Hospital Erlangen and by the patients for data privacy. We describe the recording set-up, data structure, workflow and preprocessing steps, and report the first PoCaP Corpus speech recognition analysis results with 11.52 $\%$ word error rate using pretrained models. The findings suggest that the data has the potential to build a robust command recognition system and will allow the development of a novel intervention support systems using speech and image processing in the medical domain.
BibTeX
@misc{demir2022pocap,
  author = {Kubilay Can Demir and Matthias May and Axel Schmid and Michael Uder and Katharina Breininger and Tobias Weise and Andreas Maier and Seung Hee Yang},
  title = {{PoCaP Corpus: A Multimodal Dataset for Smart Operating Room Speech Assistant using Interventional Radiology Workflow Analysis}},
  year = {2022},
  eprint = {2206.12320},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.12320},
}
Figure