A scrubbed surgeon or interventional radiologist who sees a wrong organ outline on a CT scan cannot reach for a mouse to fix it. Today they have two options. They can say "fix the liver" to a nurse who does not share their spatial picture, or they can learn engineering phrases like "shift box" and "change confidence threshold." The second option is what current language-driven segmentation tools ask for, and their own user studies report high mental demand. This proposal builds a system that accepts the complaint the clinician would naturally make, such as "the liver mask is bleeding into the gallbladder fossa." The system turns that sentence into a precise geometric statement that a program can check, generates clicks and boxes for a promptable segmentation model until the statement holds, and verifies the result against that statement rather than against a language model's opinion.
The bet is about what multi-agent task decomposition, meaning splitting one problem across several specialized language-model roles, actually adds in a sterile setting. Prior agent systems gain accuracy by separating planning from execution and by adding a critic. In surgery, though, the cost of extra language-model calls is time, and critics that argue in free text can loop without converging. We argue that the useful part of decomposition here is the intermediate artifact: a typed spatial claim that a planner writes, a grounding step attaches to real voxels, and a verifier written in ordinary code accepts or rejects. If that is right, a small planner–grounder–verifier pipeline should resolve spoken complaints in fewer turns than either a single language model emitting prompts directly or a human speaking a fixed command vocabulary.
Core Claim
Spoken correction of a 3D segmentation fails for a specific reason. The clinician's complaint is about relations between anatomical structures ("leaks into," "misses the lower tip," "includes a separate blob near the portal vein"). The promptable segmenter, however, only accepts coordinates. Language models are poor at turning one into the other directly; Beyond Lines and Circles shows spatial hallucination in LLM geometric reasoning that shrinks when spatial planning is separated from execution. Our claim is that the gap closes when the complaint is first converted into a claim in a small spatial predicate language, for example "the overlap of the current mask with structure X should be empty" or "the mask should include the connected tissue adjacent to Y in slices i through j". Each anatomical name in the claim is resolved to an automatically computed mask, and each candidate edit is accepted only if
$$\phi(M') = \text{true} \quad\text{and}\quad \frac{|(M' \,\triangle\, M) \setminus R|}{|M|} \le \varepsilon,$$
where $M$ is the current mask, $M'$ the edited mask, $\phi$ the parsed claim evaluated as code, $R$ the region the claim is about, $\triangle$ symmetric difference, and $\varepsilon$ a small tolerance for collateral change outside $R$. The second condition stops the system from "satisfying" a complaint by destroying the rest of the outline. Because acceptance is a deterministic check, the correction loop has a natural stopping rule. It ends when the check passes or when a fixed budget of segmenter calls runs out, which removes the reflection-loop deadlocks documented for critic-based agent systems in A survey on LLM-based multi-agent systems.
This differs from the closest work in specific ways. VISA uses an orchestrator and specialist agents to control a surgical console by voice, but it only navigates precomputed CT slices and toggles precomputed meshes; it never edits a segmentation. LIMIS chains a language-grounded box detector to the promptable segmenter ScribblePrompt and lets users refine results, but refinement uses fixed technical strategies that the human must choose. MACM supplies the structural template, with a Thinker that proposes conditions, a Judge that filters them, and an Executor that computes. Its Judge, however, is another language model, and its conditions are mathematical rather than spatial. SurgRAW verifies agent predictions against a knowledge graph, which works for rigid tool–action pairs. We replace that graph with geometric predicates evaluated on the actual volume.
Research Program
The system has four roles, and each role's output is inspectable. The interpreter is a locally run open language model of the size VISA used. It receives the transcribed complaint, the viewer state (current organ, slice, and plane, kept as a persistent memory in the way VISA's orchestrator does), and a list of nameable structures. It outputs a claim in the predicate language. The language has a handful of relations: exclude-overlap, include-adjacent, remove-component, fill-hole, and restrict-to-slices. It also has slots for anatomical references and slice ranges. The grounder resolves each reference to a mask from an automatic whole-body segmentation model run once before the procedure. Ground-truth masks are never used at test time, to avoid circularity. Where no automatic mask exists, for example a lesion, the grounder falls back to the current viewport and a coarse region the interpreter names. The prompt synthesizer turns the claim into prompts for the segmenter by geometry rather than by language. For exclude-overlap, it places negative clicks at the centroids of overlap components and a tightened box. For include-adjacent, it places positive scribbles along the boundary between the current mask and the missed region. It reuses LIMIS's empirical finding that slightly enlarged boxes, 10 pixels per side, stabilize ScribblePrompt decoding. The verifier evaluates the acceptance condition above. On failure, it returns a structured reason to the synthesizer, for example "overlap reduced but 4% collateral loss in slices 40–48", and the synthesizer tries again within the budget.
Speech is handled by an off-the-shelf recognizer followed by the interpreter. That puts the system in the regime VISA studied, where raw transcription accuracy of about 88.6% was raised to 95.8% task success by contextual correction. We inherit that correction step and do not claim to improve on it. Latency is a design constraint, not a headline claim. Correcting a segmentation on a planning or navigation view tolerates a few seconds, unlike the hemorrhage-time interactions that Direction 1 rightly flags, and we will report wall-clock time per resolved complaint against VISA's reported ~1.97 s per command as a reference point.
To evaluate at scale, we construct a complaint corpus on the CT datasets LIMIS used (DAP Atlas, TotalSegmentator, WORD). We corrupt ground-truth organ masks with realistic error types: leakage into a named neighbor, truncation at a pole, spurious disconnected components, and interior holes. For each corruption, clinicians write the complaint they would say, a language model paraphrases those complaints for scale, and a held-out set is spoken by clinicians and mixed with recorded operating-room noise. Because each corruption is known, every complaint has a ground-truth target mask, so success can be scored without a human in the loop. A smaller set of naturally occurring errors from an automatic segmenter supplies the real-error test.
Baselines
The naive reference point is doing nothing: the initial automatic mask with no correction. This number shows how much each method gains at all. The second reference is the LIMIS pipeline as published, Grounding DINO with LoRA feeding ScribblePrompt plus its fixed verbal refinement strategies. Here a simulated user, and in the user study a real one, chooses among those strategies. This is the strongest existing method for spoken segmentation refinement on exactly these datasets. LIMIS reports that language interactions improved Dice over the initial proposal in 65% of cases, with a box-detection mAP of 0.54. No reported result was found on Lacuna for absolute Dice after spoken refinement or for turns-to-target on these datasets, so we rerun LIMIS ourselves. We count beating it as resolving a larger fraction of complaints to within a fixed Dice of the target, within three spoken turns, while using no more segmenter calls.
The strongest competitor on our specific question is a single-model baseline. The same language model receives the complaint, the viewer state, rendered slices with the current mask and named-structure overlays, and chain-of-thought instructions, and it emits click and box coordinates for ScribblePrompt directly. It may retry after seeing the result and judging it itself. This is what a reviewer would build first, and it isolates whether the decomposition matters. It runs with the identical segmenter, backbone, inputs, prompt-tuning budget (the same number of development examples for prompt iteration), and segmenter-call budget.
Three ablations isolate the key ingredient. In the first, the code verifier is replaced by the language model judging the edited mask from rendered slices. This tests whether checkability, rather than role separation, drives gains. In the second, the grounder is removed so the interpreter must name coordinates itself. In the third, the language model is used only to parse the claim, and prompts are found by blind randomized search over clicks and boxes in region $R$ until the verifier passes. This tests whether the planner's geometric synthesis earns its keep or whether verification alone does the work.
Why not just…?
Why not just let one language model read the complaint and output the clicks? This is the most direct threat. Evidence from geometry and from agent decomposition (Beyond Lines and Circles; MACM's rise from 54.68% to 76.73% on Level 5 MATH over single-prompt GPT-4 Turbo) suggests single-thread spatial output is unreliable. Those are different tasks, though, and none of that evidence is about 3D masks. The single-model baseline above, given equal inputs and budget, settles it directly. If it matches our pipeline, the honest conclusion is that a checkable claim is unnecessary, and the paper becomes a negative result about decomposition here.
Why not just give clinicians a richer fixed command set, as in LIMIS? Fixed commands are predictable, and predictability matters in the operating room. LIMIS's own user study, however, reports high NASA-TLX mental demand when clinicians must express spatial fixes in its technical vocabulary. The comparison that matters is turns and workload at equal accuracy. Our user study measures both on the same cases, and if fixed commands reach targets in as few turns with similar workload, the free-form interpreter is not justified.
Why not just search over prompts until a checker passes, with no planner? This could plausibly work as well, because once a code verifier exists, brute force is a legitimate solver. That is why the blind-search ablation is part of the core design rather than an afterthought. We expect search to match accuracy only at many more segmenter calls, and therefore more latency. If it matches at comparable cost, the contribution narrows to the predicate language and verifier, and we will report it that way.
Why not just use a stronger automatic or text-promptable segmenter so corrections are rarely needed? Better segmenters such as SegVol or SAM 2 for 3D reduce error rates but do not remove them. LIMIS reports that fully automatic models fail under atypical anatomy and pathology, and the 2018 scene segmentation challenge found occluded tissue at 0.224 mIoU. Our pipeline is segmenter-agnostic, so we will run it on top of a stronger base model as well. The question is whether residual errors still benefit from correction, not whether the base model improves.
Proposed Experiments
The main experiment runs all methods on the synthetic-corruption corpus from the three CT datasets and reports four quantities: the fraction of complaints resolved, meaning the edited mask is within a fixed Dice of the target; Dice after one, two, and three turns; segmenter calls and wall-clock time per resolved complaint; and collateral change outside $R$. Results are split by error type and by whether the referenced structure has an automatic mask. We expect the method to be strongest on leakage into named neighbors and weakest on lesions and unnamed regions, where grounding falls back to the viewport. Reporting that split is part of the claim, not a footnote.
A noise experiment replays the spoken subset through the recognizer with and without operating-room noise. It reports how much transcription error survives into wrong claims, and whether the verifier catches claims that are well formed but wrong. It cannot catch a misheard organ name that still parses, so we will count those cases explicitly. A reflection-convergence analysis compares how often each method exhausts its budget without terminating, testing whether code verification removes the non-convergent critique loops seen with language-model critics.
A clinician user study, run with draped hands and in a simulated noise environment, compares our system with LIMIS strategies on matched real-error cases. It measures turns to acceptance, time, and NASA-TLX.
The decisive checks are three. First, the method must beat the single-model baseline at equal segmenter-call budget on resolution rate; otherwise decomposition is not doing the work. Second, removing the code verifier, the language-model-judge ablation, must hurt; otherwise checkability is not the ingredient. Third, the planner must reach the verifier's acceptance in fewer segmenter calls than blind search; otherwise the language model's only job is parsing. Passing all three would show that, for sterile spoken interaction with 3D segmentations, the value of multi-agent decomposition lies in a checkable spatial claim rather than in debate.
References
- Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Models
- A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges
- Voice-Interactive Surgical Agent for Multimodal Patient Data Control
- LIMIS: Towards Language-Based Interactive Medical Image Segmentation
- MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems
- SurgRAW: Multi-Agent Workflow With Chain of Thought Reasoning for Robotic Surgical Video Analysis
- SegVol: Universal and Interactive Volumetric Medical Image Segmentation
- Interactive 3D Medical Image Segmentation with SAM 2
- 2018 Robotic Scene Segmentation Challenge