Modern clinical workflows are heavily dependent on digital data. Surgeons and radiologists constantly need to interact with real-time fluoroscopy, 3D tomographic models, electronic health records, and automated segmentation software. However, a fundamental tension exists between the digital world and the operating room: traditional input devices—such as mice, keyboards, and touchscreens—are incompatible with strict sterility protocols.
As noted by Zhou et al. in Touchless Telerobotic Surgery - Is It Possible at All?, physical input devices act as dangerous vectors for bacterial transmission. When a clinician needs to adjust an image or input data during a procedure, they must either break sterility, use cumbersome plastic drapes, or verbally instruct a non-sterile assistant to perform the task. This last option introduces significant communication lag and cognitive friction. Furthermore, as Weigand et al. point out in A concept and technical requirements for the Temi platform supporting care and nursing, a clinician's hands are frequently occupied by surgical instruments or direct patient care, rendering tactile interfaces physically inaccessible.
To overcome these aseptic constraints, researchers have developed interface paradigms that allow seamless digital interaction without compromising the sterile field. These solutions fall into two primary categories: touchless interaction (leveraging voice and mid-air gestures) and consolidated single-operator tactile controls (re-engineering physical hardware to minimize contamination and coordination overhead).
Touchless Interaction via Voice and Natural Language
Voice control provides an ideal mechanism for hands-free interaction because it completely decouples the clinician's physical state from the digital system. A surgeon can issue commands while their hands remain scrubbed and actively engaged with surgical instruments.
Early voice systems were limited to rigid, discrete commands, but recent breakthroughs leverage deep learning and visual language models (VLMs) to interpret contextual intent. For example, in SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imaging, Pham et al. developed an end-to-end speech-driven VLM designed specifically for operating rooms. This allows physicians to query diagnostic tools hands-free and receive structured, explainable reasoning about medical abnormalities in real time.
[Spoken Intention] ──> [Audio/Language Model] ──> [Contextual Software Action]
│ │
└─ (Hands remain sterile & occupied) └─ (No physical contact required)
This paradigm is particularly powerful for complex, iterative tasks like medical image segmentation, which traditionally require meticulous mouse clicks. To bypass this requirement, Jaus et al. introduced LIMIS: Towards Language-based Interactive Medical Image Segmentation. The framework allows clinicians to generate and refine organ masks purely through natural language prompts, translating verbal feedback into precise spatial modifications.
To further minimize cognitive load during surgery, Zhang et al. developed ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding. This system uses audio cues to understand the surgeon's immediate intent, automatically isolating and segmenting relevant surgical tools on the display without manual intervention.
The utility of voice interfaces extends beyond the operating theater into other high-activity, sterile environments:
- Wet Labs: Lou et al. developed Gene Tracer: a smart, interactive, voice-controlled Alexa skill For gene information retrieval and browsing, mutation annotation and network visualization, enabling molecular biologists to query genomic databases hands-free while handling sensitive biological samples.
- Diagnostic Radiology: In Speech-based Medical Decision Support in VR using a Deep Neural Network (Demonstration), Prange et al. demonstrated a multimodal virtual reality system where radiologists navigate complex patient records and retrieve therapy recommendations using natural speech.
Contactless Gesture Tracking and Spatial Computing
While voice control excels at semantic queries and discrete commands, it struggles with spatial manipulation, such as rotating a 3D anatomical model or positioning a virtual probe. To bridge this gap, contactless gesture tracking uses computer vision and spatial computing to map mid-air hand movements directly to user interface actions, maintaining a physical barrier of air between the clinician and the machine.
In Interactive Planning of Cryotherapy Using Physics-Based Simulation, Lekkal et al. integrated a Leap Motion sensor into pre-operative planning software. This configuration allows surgeons to manipulate virtual cryoprobes and medical images using intuitive hand gestures, eliminating the need to step away from the sterile field to use a mouse.
Extended reality (XR) headsets take this a step further by projecting data directly into the user's line of sight. Jang et al. utilized the Microsoft HoloLens in Three-dimensional holographic visualization of high-resolution myocardial scar on HoloLens to render interactive, 3D holograms of myocardial scars. By combining voice commands with mid-air gestures, the primary operator retains direct, unassisted control over data visualization.
[Depth/Laser Sensors] ──> [Point Cloud / Geometry Engine] ──> [Spatial UI Manipulation]
│ │
└─ (Tracks mid-air hand movements) └─ (Rotates 3D models, positions probes)
As surgical robotics become more prevalent, touchless interaction is transitioning from an optional convenience to a core architectural requirement. Żelechowski et al. argue in Automatic patient positioning based on robot rotational workspace for extended reality that traditional handheld controllers are fundamentally unsuited for robotic surgical setup. They advocate for touchless hand tracking to maintain aseptic conditions during the critical phases of patient and robot positioning.
To ensure these systems are reliable enough for clinical deployment, the underlying computer vision must be exceptionally robust to variations in lighting and hand orientation. Addressing this need, Xing et al. introduced a 3D deep learning model in Intelligent rehabilitation in an aging population: empowering human-machine interaction for hand function rehabilitation through 3D deep learning and point cloud. By processing raw point cloud data from laser sensors, their system achieves highly accurate, non-contact gesture recognition, demonstrating how robust spatial tracking can mitigate infection risks while preserving intuitive human-machine interaction.
Consolidated Single-Operator Tactile Controls
Touchless interfaces are highly effective for software navigation, but certain clinical tasks inherently require physical contact with specialized medical hardware, such as endoscopes, catheters, or imaging consoles. In these scenarios, the risk of contamination and workflow disruption is mitigated by ergonomic control consolidation.
Traditional interventional systems often require a multi-person team to coordinate a single procedure (e.g., one clinician steering an endoscope while an assistant manipulates a secondary catheter or operates an imaging console). This division of labor introduces communication overhead and increases the number of physical contact points that must be kept sterile. Consolidating these functions into a unified, single-operator interface allows a single scrubbed clinician to manage the entire workflow, keeping their eyes on the task and minimizing the surface area of potential contamination.
Traditional Multi-Person Workflow:
[Clinician A (Primary Tool)] <──(Verbal Coordination Lag)──> [Clinician B (Console/Secondary Tool)]
Consolidated Single-Operator Workflow:
[Single Clinician] ──> [Unified Tactile Interface (Joystick/Integrated Controls)] ──> [Simultaneous Control]
This design philosophy dates back to early digital interventional radiology suites. Uzuyama et al. demonstrated this concept in Development of Digital Image Controller for IVR and Development of Image Controller of DIGITEX2400 for IVR. By mapping complex digital image playback, processing, and measurement functions to a single-handed joystick equipped with integrated push buttons, the authors allowed interventional radiologists to manipulate real-time visual data with one hand. This eliminated the need to look away from the patient or interact with standard, non-sterile keyboards and mice.
In modern interventional endoscopy, control consolidation has enabled highly complex procedures to be performed by a single operator:
- Laser Lithotripsy: In Efficacy and safety of novel digital single-operator peroral cholangioscopy-guided laser lithotripsy for complicated biliary stones, Wong evaluated a next-generation digital cholangioscope that integrates steering and catheter controls. This single-operator architecture allows a single clinician to simultaneously manage the primary endoscope and the specialized laser catheter, significantly streamlining the clearance of complicated biliary stones.
- Gallbladder Drainage: Similarly, Ong et al. demonstrated the advantages of unified hardware design in Endoscopic ultrasound-guided gallbladder drainage using a novel cautery-enhanced, endoscopist-deployed lumen-apposing metal stent. By consolidating the heating element and stent deployment mechanisms into a single-operator interface, the system removes the need for a coordinated multi-person deployment technique. This integration minimizes communication errors, accelerates the procedure, and reduces the logistical complexity of maintaining a sterile field.
Key Foundational Work
194. Development of Digital Image Controller for IVR (1994) — contributed 3 concept elements to this direction.
Real-Time Range Imaging in Health Care: A Survey (2013) — contributed 2 concept elements to this direction.
(8) Development of Image Controller of DIGITEX2400 for IVR (1994) — contributed 2 concept elements to this direction.