TL;DR
VISA (Voice-Interactive Surgical Agent) is a hierarchical multi-agent framework that allows robotic surgeons to access and manipulate multimodal patient data—like CT scans and 3D anatomical models—using natural, free-form voice commands. By leveraging Large Language Models (LLMs) for orchestration and error correction, the system achieves a 95.8% success rate in executing complex surgical tasks without requiring the surgeon to take their hands off the controls.
What Problem Does This Paper Address?
In the high-stakes environment of robotic surgery (e.g., using the da Vinci system), surgeons are "eyes-on, hands-on." Their hands are occupied with manipulators, and their vision is locked into a 3D console.
When a surgeon needs to check a patient’s preoperative CT scan, verify a pulmonary function test (PFT), or rotate a 3D model of a tumor, they typically have to pause the procedure, look away from the console, or rely on an assistant to navigate a separate monitor. Existing voice-control systems in the OR are often rigid, relying on a narrow set of predefined "magic words" that fail if the surgeon hesitates or if the speech-to-text (STT) engine misinterprets medical jargon.
Methods
The authors propose VISA, a hierarchical Multi-Agent System (MAS) driven by LLMs (specifically Gemma3 27B). The architecture is split into two main layers:
- Workflow Orchestrator Agent: This acts as the "brain," using probabilistic decision-making to plan the sequence of functions (e.g., transcribing audio, validating the command, and routing it to the right specialist).
- Task-Specific Agents:
- Information Retrieval (IR): Overlays clinical data (age, diagnosis, lab results) onto the surgical video.
- Image Viewer (IV): Navigates 2D CT slices (axial, coronal, sagittal) with commands like "move to the middle slice."
- Anatomy Rendering (AR): Manipulates 3D models, allowing for rotation, zooming, and toggling specific structures (e.g., "hide the left lung").
The "Secret Sauce": VISA doesn't just execute commands; it reasons through them. It includes a "Correct and Validate" stage where the LLM fixes common STT errors (e.g., correcting "city" to "CT") and a "Command Reasoning" stage that resolves linguistic ambiguity based on the current "Memory State" of the surgery.
Datasets
The study introduced a new benchmark for this niche:
- Command Dataset: 240 user commands categorized by structure (single vs. composite), type (explicit vs. implicit), and expression (baseline vs. paraphrase).
- Multimodal Data: Real patient data including surgical videos, CT DICOM images, and 3D models generated via TotalSegmentator.
- Voice Variability: To test robustness, the authors used four different synthesized neural voices (Microsoft Edge TTS) and real human speech to simulate various accents and tones found in a global OR.
Key Insights and Ideas
- LLMs as Error Correctors: One of the biggest "aha" moments is using the LLM to bridge the gap between imperfect STT and execution. If the STT transcribes "age info" as "Age in full," the LLM recognizes the intent and maps it to the correct data column.
- Context Awareness: Because the system maintains a "Global Memory State," it understands implicit commands. If a surgeon says "Zoom in" after looking at a CT, the IV agent takes over. If they say it while looking at a 3D model, the AR agent handles it.
- Handling Composite Tasks: Unlike older systems, VISA can handle multi-step instructions like "Move axial to the middle slice and zoom in" by decomposing them into sequential actions.
- Zero-Shot Medical Mapping: The LLM's internal knowledge allows it to understand that "How old is the patient?" is a request for the "Age" field without explicit hard-coding for every possible question.
Key Results
The authors evaluated VISA using a new metric called MOEM (Multi-level Orchestration Evaluation Metric):
- Robustness: While the raw STT accuracy was only ~88.6% (due to "davinci" wake-word misidentification), the Multi-pass Success Rate reached 95.8%. This means the system successfully recovered from initial errors through its correction and reasoning loops.
- Task Performance: The IR agent was the most accurate (97.7%), while the IV and AR agents showed slightly lower but still high performance (93.2% and 93.9% respectively) due to the higher complexity of spatial commands.
- Efficiency: The system runs locally on an NVIDIA RTX 4500 Ada GPU, maintaining a latency of ~1.97s for recognition, which is viable for non-emergency data retrieval during surgery.
Limitations and Future Work
- Latency: While ~2 seconds is fast, the cumulative delay of multiple sequential LLM calls can be a bottleneck in time-sensitive environments. The authors suggest future work on "caching reasoning patterns" to speed this up.
- Dataset Scale: 240 commands is a strong start, but real-world clinical phrasing is infinitely variable. Future iterations will need larger, multi-institutional datasets.
- Customization: Different surgical specialties (e.g., neurosurgery vs. thoracic) require different data views. The authors aim to develop a "self-evolving" orchestration mechanism that learns a specific surgeon's preferences over time.
Summary updated