TL;DR
The PoCaP Corpus is a novel multimodal dataset designed to pave the way for voice-controlled smart assistants in the operating room. By combining German audio recordings, X-ray images, and device system logs from 31 real-world surgeries, the dataset enables researchers to analyze surgical workflows and train robust speech recognition models tailored to the medical domain.
What Problem Does This Paper Address?
Modern operating rooms (ORs) are highly digitalized, packed with advanced medical devices that generate massive amounts of data. While this technology improves patient safety, it also increases cognitive load for surgeons, who must split their attention between the patient and various equipment interfaces.
To alleviate this, researchers want to build intelligent workflow assistance systems—like a hands-free, voice-controlled smart assistant that can adjust an operating table or move a C-arm X-ray machine on command. However, developing these systems requires high-quality data. While plenty of surgical datasets exist, they consist almost exclusively of endoscopic or microscopic video. Modalities like speech, which are universal to almost every surgery, remain severely under-investigated due to the difficulty of recording in high-stress, sterile environments and strict data privacy laws.
Methods
The authors developed a synchronized, multi-channel data collection pipeline during 31 Port-Catheter Placement (PoCaP) interventions.
The setup captured three distinct data streams:
- Audio/Speech: The primary surgeon and assistant wore high-quality wireless headsets to capture clear, two-channel verbal communication. A third ambient audio channel was captured via a GoPro camera.
- X-Ray Images: Instead of pulling raw medical files post-op, the team used Open Broadcaster Software (OBS) to duplicate and record the live widescreen display feed in the OR, saving frames at 1 frame per second (fps).
- System Commands: A software called ACSOS monitored the local network, generating a "ticket" (log file) every time a C-Arm machine function was used, tracking up to 393 different variables like physical rotation angle and pulse rate.
Because these devices were started manually, the authors used the autocorrelation of the audio channels to precisely align the ambient video with the headset audio, and used visual cues (like the launch screen of the logging software on the recorded display) to sync the system commands. Voice Activity Detection (VAD) and pre-trained Automatic Speech Recognition (ASR) models were then applied to transcribe the conversations.
Datasets
The paper introduces the PoCaP (Port Catheter Placement) Corpus. It contains data from 31 interventions performed by six different surgeons, with an average operation duration of roughly 81 minutes. The dataset is hierarchically annotated into 8 surgical phases and 31 distinct surgical steps.
Unlike well-known open datasets in the field—such as Cholec80 or HeiChole (which focus on laparoscopic gallbladder removals via endoscopic video) or CATARACTS (microscopic eye surgery video)—the PoCaP Corpus introduces speech and device logs. Due to strict institutional review board (IRB) and patient privacy constraints, this dataset cannot be made publicly available, but it serves as a blueprint for multimodal medical data collection.
Key Insights and Ideas
- X-Rays as Workflow Landmarks: In interventional radiology, X-rays are used sparingly to minimize radiation doses to the patient. Because they are only taken during specific, critical moments (e.g., verifying guide wire or catheter positioning), the appearance of an X-ray image acts as a highly reliable "landmark" for identifying the current phase of a surgery.
- The Challenge of "OR Noise": Not all speech in an OR is a command. Surgeons chat with assistants, discuss unrelated medical topics, or use highly localized dialects. A smart assistant must not only transcribe words accurately but also semantically filter out non-intervention-related conversations.
- Speech is Highly Scalable: Unlike video data, which changes drastically depending on whether a surgery is open, laparoscopic, or microscopic, speech is a constant across all surgical domains, making it an ideal modality for generalized OR assistance.
Key Results
To find the best baseline for transcribing medical German in a noisy environment, the authors benchmarked six pre-trained ASR models on a manually transcribed subset of their data.
While standard models struggled heavily—one common model hit a disastrous 85.80% Word Error Rate (WER)—the stt_de_conformer_transducer_large model performed exceptionally well, achieving a 11.52% WER. This low error rate demonstrates that pre-trained, non-medical models can still be highly robust against medical terminology and ambient OR noise, proving the feasibility of speech-controlled systems.
Limitations and Future Work
The primary limitation of the PoCaP Corpus is its scale and localization: it features 31 interventions from a single medical center (University Hospital Erlangen) involving just six surgeons.
Moving forward, the authors plan to expand the dataset to include more surgeries and diverse environments. Future algorithmic work will focus on translating medical dialects into standard German, mapping synonyms (different phrases used by surgeons to mean the same command), and training machine learning models to automatically predict the remaining time in a surgery based on the live multimodal stream.
Summary updated