Voice-Interactive Surgical Agent for Multimodal Patient Data Control

Hyeryun Park, Byung Mo Gu, Jun Hee Lee, Byeong Hyeon Choi, Sekeun Kim, Hyun Koo Kim, Kyungsang KimarXiv 2025 · DOI · arXiv

TL;DR

VISA (Voice-Interactive Surgical Agent) is a hierarchical multi-agent framework that allows robotic surgeons to access and manipulate multimodal patient data—like CT scans and 3D anatomical models—using natural, free-form voice commands. By leveraging Large Language Models (LLMs) for orchestration and error correction, the system achieves a 95.8% success rate in executing complex surgical tasks without requiring the surgeon to take their hands off the controls.

What Problem Does This Paper Address?

In the high-stakes environment of robotic surgery (e.g., using the da Vinci system), surgeons are "eyes-on, hands-on." Their hands are occupied with manipulators, and their vision is locked into a 3D console.

When a surgeon needs to check a patient’s preoperative CT scan, verify a pulmonary function test (PFT), or rotate a 3D model of a tumor, they typically have to pause the procedure, look away from the console, or rely on an assistant to navigate a separate monitor. Existing voice-control systems in the OR are often rigid, relying on a narrow set of predefined "magic words" that fail if the surgeon hesitates or if the speech-to-text (STT) engine misinterprets medical jargon.

Methods

The authors propose VISA, a hierarchical Multi-Agent System (MAS) driven by LLMs (specifically Gemma3 27B). The architecture is split into two main layers:

  1. Workflow Orchestrator Agent: This acts as the "brain," using probabilistic decision-making to plan the sequence of functions (e.g., transcribing audio, validating the command, and routing it to the right specialist).
  2. Task-Specific Agents:
    • Information Retrieval (IR): Overlays clinical data (age, diagnosis, lab results) onto the surgical video.
  3. Image Viewer (IV): Navigates 2D CT slices (axial, coronal, sagittal) with commands like "move to the middle slice."
    • Anatomy Rendering (AR): Manipulates 3D models, allowing for rotation, zooming, and toggling specific structures (e.g., "hide the left lung").

The "Secret Sauce": VISA doesn't just execute commands; it reasons through them. It includes a "Correct and Validate" stage where the LLM fixes common STT errors (e.g., correcting "city" to "CT") and a "Command Reasoning" stage that resolves linguistic ambiguity based on the current "Memory State" of the surgery.

Datasets

The study introduced a new benchmark for this niche:

  • Command Dataset: 240 user commands categorized by structure (single vs. composite), type (explicit vs. implicit), and expression (baseline vs. paraphrase).
  • Multimodal Data: Real patient data including surgical videos, CT DICOM images, and 3D models generated via TotalSegmentator.
  • Voice Variability: To test robustness, the authors used four different synthesized neural voices (Microsoft Edge TTS) and real human speech to simulate various accents and tones found in a global OR.

Key Insights and Ideas

  • LLMs as Error Correctors: One of the biggest "aha" moments is using the LLM to bridge the gap between imperfect STT and execution. If the STT transcribes "age info" as "Age in full," the LLM recognizes the intent and maps it to the correct data column.
  • Context Awareness: Because the system maintains a "Global Memory State," it understands implicit commands. If a surgeon says "Zoom in" after looking at a CT, the IV agent takes over. If they say it while looking at a 3D model, the AR agent handles it.
  • Handling Composite Tasks: Unlike older systems, VISA can handle multi-step instructions like "Move axial to the middle slice and zoom in" by decomposing them into sequential actions.
  • Zero-Shot Medical Mapping: The LLM's internal knowledge allows it to understand that "How old is the patient?" is a request for the "Age" field without explicit hard-coding for every possible question.

Key Results

The authors evaluated VISA using a new metric called MOEM (Multi-level Orchestration Evaluation Metric):

  • Robustness: While the raw STT accuracy was only ~88.6% (due to "davinci" wake-word misidentification), the Multi-pass Success Rate reached 95.8%. This means the system successfully recovered from initial errors through its correction and reasoning loops.
  • Task Performance: The IR agent was the most accurate (97.7%), while the IV and AR agents showed slightly lower but still high performance (93.2% and 93.9% respectively) due to the higher complexity of spatial commands.
  • Efficiency: The system runs locally on an NVIDIA RTX 4500 Ada GPU, maintaining a latency of ~1.97s for recognition, which is viable for non-emergency data retrieval during surgery.

Limitations and Future Work

  • Latency: While ~2 seconds is fast, the cumulative delay of multiple sequential LLM calls can be a bottleneck in time-sensitive environments. The authors suggest future work on "caching reasoning patterns" to speed this up.
  • Dataset Scale: 240 commands is a strong start, but real-world clinical phrasing is infinitely variable. Future iterations will need larger, multi-institutional datasets.
  • Customization: Different surgical specialties (e.g., neurosurgery vs. thoracic) require different data views. The authors aim to develop a "self-evolving" orchestration mechanism that learns a specific surgeon's preferences over time.

Summary updated

Abstract

In robotic surgery, surgeons fully engage their hands and visual attention in procedures, making it difficult to access and manipulate multimodal patient data without interrupting the workflow. To overcome this problem, we propose a Voice-Interactive Surgical Agent (VISA) built on a hierarchical multi-agent framework consisting of an orchestration agent and three task-specific agents driven by Large Language Models (LLMs). These LLM-based agents autonomously plan, refine, validate, and reason to interpret voice commands and execute tasks such as retrieving clinical information, manipulating CT scans, or navigating 3D anatomical models within surgical video. We construct a dataset of 240 user commands organized into hierarchical categories and introduce the Multi-level Orchestration Evaluation Metric (MOEM) that evaluates the performance and robustness at both the command and category levels. Experimental results demonstrate that VISA achieves high stage-level accuracy and workflow-level success rates, while also enhancing its robustness by correcting transcription errors, resolving linguistic ambiguity, and interpreting diverse free-form expressions. These findings highlight the strong potential of VISA to support robotic surgery and its scalability for integrating new functions and agents.
BibTeX
@misc{park2025voice,
  author = {Hyeryun Park and Byung Mo Gu and Jun Hee Lee and Byeong Hyeon Choi and Sekeun Kim and Hyun Koo Kim and Kyungsang Kim},
  title = {{Voice-Interactive Surgical Agent for Multimodal Patient Data Control}},
  year = {2025},
  eprint = {2511.07392},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.07392},
}
Figure