Refining 3D Medical Segmentation with Verbal Instruction

Kangxian Xie, Jiancheng Yang, Nándor Pintér, Chao Wu, Behzad Bozorgtabar, Mingchen Gao

In the world of medical imaging, 3D segmentation—the process of digitally isolating anatomical structures like blood vessels—is a cornerstone of surgical planning and diagnosis. But even the most advanced AI models frequently stumble, producing "broken" vessels or incorrect thicknesses due to variations in patient populations or scanner protocols. When this happens, a radiologist can see the error instantly, but fixing it usually requires tedious, manual voxel-by-voxel editing.

A new paper, "Refining 3D Medical Segmentation with Verbal Instruction," proposes a more intuitive "clinician-in-the-loop" approach. Instead of clicking and dragging, what if a doctor could simply tell the computer: "The lumen is disconnected near the proximal end," and the AI fixed it?

TL;DR

The authors introduce CoWTalk, the first benchmark for instruction-guided 3D medical shape refinement, and a novel deep learning framework that "listens" to verbal instructions to iteratively fix segmentation errors. Their model significantly outperforms standard segmentation baselines and general-purpose shape editors, proving that natural language is a viable tool for precise anatomical modeling.

What Problem Does This Paper Address?

Automated 3D segmentation models often produce suboptimal results. They might miss a small vessel segment, create "fragmented" structures, or get the diameter of an artery wrong. While radiologists have a clear mental model of the correct anatomy, there hasn't been an efficient way to translate their verbal expertise into geometric corrections.

The primary bottleneck has been data scarcity. To train a model to follow instructions, you need a dataset consisting of:

  1. An erroneous 3D shape.
  2. A natural language instruction describing the fix.
  3. The "ground truth" (correct) 3D shape.

Such a dataset didn't exist for the medical domain—until now.

Methods

The researchers developed an end-to-end vision-language framework with an encode-fusion-decode architecture:

  • Shape Encoding (Vector Sets): Instead of processing massive, clunky 3D voxel grids, the model represents the medical structure as a set of "shape latents" (vector sets). This makes the representation more flexible for editing.
  • Text Encoding: They use a pre-trained BERT model to convert the radiologist's instructions into high-dimensional text features.
  • Multi-modal Interaction: A transformer-based encoder allows the shape latents to "attend" to the text. This is where the magic happens: the model identifies which parts of the 3D geometry correspond to the words in the instruction (e.g., linking the word "thicken" to a specific arterial branch).
  • Implicit Point-wise Decoding: To ensure the final output is smooth and anatomically plausible, the model uses an implicit function to predict the final shape. This allows for continuous, high-resolution surfaces rather than jagged voxels.
  • Iterative Refinement: The process is designed to be iterative. A user can give multiple instructions in sequence to progressively "sculpt" the final, accurate anatomy.

Datasets

The authors introduced a new benchmark called CoWTalk, built upon the existing TopCoW dataset (which focuses on the Circle of Willis, the complex web of arteries in the brain).

  • Synthesis: They generated 3,750 samples by taking perfect anatomical models and "breaking" them using controlled morphological perturbations (e.g., thinning, shortening, or fragmenting segments).
  • Instructions: They used templates to describe these errors and then employed ChatGPT to rephrase them into realistic, radiologist-style narratives (e.g., "BA is shortened near its bottom end... Extend it").

Key Insights and Ideas

  • Language as a Precision Tool: The paper demonstrates that language isn't just for "vibes" or general style; it can provide precise geometric guidance for complex tubular structures like arteries.
  • Vector Sets > Voxels: By using vector sets (inspired by 3DS2VS), the model avoids the memory overhead of high-resolution 3D grids while remaining sensitive to fine-grained local edits.
  • The Power of Iteration: Breaking a complex repair into smaller, iterative steps significantly improves the final Dice score (a measure of overlap with the ground truth). The model performs better when it can "focus" on one correction at a time.
  • Image-Optional, but Image-Enhanced: The model can refine shapes using only the 3D mask and the text, but it performs even better when it has access to the original medical image (MRI/CT) as additional context.

Key Results

The proposed method showed impressive gains across several metrics:

  • Quantitative: On the CoWTalk benchmark, the model achieved an 87.5% Dice score using only shape and text—a 6.8% improvement over the "broken" input.
  • Beating the Giants: When the medical image was included, the model reached a 91.8% Dice score, outperforming established state-of-the-art segmentation models like nnUNetv2 (90.0%) and SwinUNetrv2 (84.7%).
  • Qualitative: Visual results show the model successfully "re-connecting" fragmented vessels and "extending" shortened segments based solely on text prompts, tasks that traditional segmentation models often fail to correct.

Limitations and Future Work

While a major step forward, the authors note a few areas for growth:

  • Synthetic vs. Real Errors: CoWTalk uses synthesized errors. While these are mathematically rigorous, real-world segmentation errors might be more chaotic or subtle.
  • Instruction Complexity: The current model handles direct anatomical corrections. Future versions could potentially handle higher-level clinical reasoning (e.g., "Adjust the segmentation to match the contrast flow in the late phase").
  • Real-time Interaction: The paper focuses on the feasibility of the architecture; optimizing it for real-time, "live" conversation with a radiologist in a clinical suite remains a future engineering challenge.

Paper Citation: Xie, K., Yang, J., Pinter, N., Wu, C., Bozorgtabar, B., & Gao, M. (2026). Refining 3D Medical Segmentation with Verbal Instruction. arXiv:2603.14496v1 [cs.CV].

Summary updated

Abstract

Accurate 3D anatomical segmentation is essential for clinical diagnosis and surgical planning. However, automated models frequently generate suboptimal shape predictions due to factors such as limited and imbalanced training data, inadequate labeling quality, and distribution shifts between training and deployment settings. A natural solution is to iteratively refine the predicted shape based on the radiologists' verbal instructions. However, this is hindered by the scarcity of paired data that explicitly links erroneous shapes to corresponding corrective instructions. As an initial step toward addressing this limitation, we introduce CoWTalk, a benchmark comprising 3D arterial anatomies with controllable synthesized anatomical errors and their corresponding repairing instructions. Building on this benchmark, we further propose an iterative refinement model that represents 3D shapes as vector sets and interacts with textual instructions to progressively update the target shape. Experimental results demonstrate that our method achieves significant improvements over corrupted inputs and competitive baselines, highlighting the feasibility of language-driven clinician-in-the-loop refinement for 3D medical shapes modeling.
BibTeX
@misc{xie2026refining,
  author = {Kangxian Xie and Jiancheng Yang and Nándor Pintér and Chao Wu and Behzad Bozorgtabar and Mingchen Gao},
  title = {{Refining 3D Medical Segmentation with Verbal Instruction}},
  year = {2026},
  eprint = {2603.14496},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.14496},
}

Loading related content...

Figure