NativeSeg: Native Vision Language Segmentation of 3D Computed Tomography via Spatial Visual Tokens and Verified Local Zoom

Chengqian Ma, Gaole Dai, Zijie Meng, Weixuan Jin, Yijie Du, Yuexiao Ma, Xiawu Zheng, Fei ChaoarXiv 2026 · arXiv

TL;DR

NativeSeg is a "native" vision-language framework for 3D CT segmentation that eliminates the need for external decoders or lossy codebooks. By performing dense mask decoding, volumetric triage, and self-diagnostic local zooming directly over its internal 2D spatial tokens, it achieves state-of-the-art precision and robust zero-shot performance on pediatric data.

What Problem Does This Paper Address?

Segmenting organs in 3D Computed Tomography (CT) is a balancing act. You need a global view to identify the organ and distinguish it from neighbors, but you also need high-resolution local views to trace fine boundaries, like the neck of a gallbladder.

Most current Medical Vision-Language Models (VLMs) are "non-native" segmenters. They act as symbolic routers: they might reason about the image but then delegate the actual pixel-level work to an external tool like the Segment Anything Model (SAM) using a single 1D [SEG] token. This creates a "geometric bottleneck" where complex 2D shapes are compressed into a single vector, losing vital spatial information. Furthermore, in a typical 3D CT volume, over 80% of slices don't even contain the target organ. Existing models often lack efficient "triage" to skip these empty slices, leading to wasted computation and false positives.

Methods

NativeSeg transforms a large language model (specifically a LoRA-adapted Qwen3.8-27B) into a native segmenter through several novel components:

  • Native 2D Spatial Token Readout: Instead of compressing reasoning into one token, NativeSeg preserves the VLM’s internal 16×16 spatial token grid. These tokens are fused with a lightweight Five-Slice U-Net decoder that incorporates local 3D context.
  • Volumetric Triage via Presence Gate (<pres>): The model includes a learnable token that predicts if the target organ is even present in the slice. If the probability is low, it skips the slice entirely.
  • Self-Diagnosis via Error Gate (<err>): A multiscale error token inspects the model's own mask. It predicts misclassified areas across a grid, identifying exactly where the boundaries are uncertain.
  • Verified Local Zoom: If the error gate detects high uncertainty, the model "zooms" into that specific region of interest (ROI), re-encodes it at higher resolution, and generates a refined mask. Crucially, this is a closed-loop process: the model only accepts the update if the predicted error actually decreases.
Figure 2. Architecture and native volumetric pipeline of NativeSeg. (a) Given the five slice stack (1a) and global or zoomed overlay views (1b), the frozen vision tower (2) encodes CT views while the LoRA adapted Qwen3.8-27B backbone (3) processes five native token groups without external SAM heads: organ Prompt (q), Global and Local 2D spatial visual tokens, <pres>, and <err>. Upper layer states drive dense spatial readout (4a), empty slice presence screening (4b), and 20 cell multiscale error diagnosis (4c) to halt when predicted error is small (Eq. (2)) or zoom into uncertain boundaries (Eq. (3)) with closed loop verification (Steps 5a to 5c). (b to g) Multiround gallbladder refinement example.

Datasets

The authors trained and evaluated NativeSeg on:

  • TotalSegmentator: 1,081 adult CT scans used for training and in-domain evaluation.
  • Pediatric-CT-SEG: 21 pediatric patients used for zero-shot out-of-distribution evaluation. This is a significant challenge due to the different body scales and visceral fat contrasts in children.

The evaluation covered seven abdominal organs: spleen, right/left kidneys, gallbladder, liver, stomach, and pancreas.

Key Insights and Ideas

  • Spatiality is Key: The "aha moment" here is that the spatial topology already computed inside the VLM is far more valuable for segmentation than a compressed 1D embedding. Moving from a [SEG] token to a spatial token grid provided a massive +7.04 point boost in Dice Similarity Coefficient (DSC).
  • VLMs as Self-Critics: The paper demonstrates that a VLM can be trained to recognize its own failures. By rendering its current mask back onto the image and "looking" at it with the <err> token, the model performs a native self-critique that rivals external specialist networks.
  • Efficiency through Triage: By halting at "zero zooms" on over 91% of slices (those that are either empty or already well-segmented), NativeSeg remains computationally viable despite its large backbone.
  • Plug-and-Play Refinement: NativeSeg isn't just a standalone model; it can act as a "safety filter" for other specialists (like nnU-Net). It can verify and refine their outputs, only making changes when it's confident the error will drop.
Figure 1. Vision language medical segmentation paradigms on an abdominal computed tomography (CT) right kidney slice (a to d) and volumetric triage with verified zoom on adult CT (e). (a) Single pass 1D segmentation token ([SEG]) broadcast into an external Segment Anything Model (SAM) decoder; (b) discrete vector quantized (VQ) codebook lookup; (c) stateless box prompted SAM refinement; (d) NativeSeg (Ours), decoding confidence maps from 2D spatial tokens in a five slice U-Net decoder with verified region of interest (ROI) zoom to raise slice Dice similarity coefficient (DSC); (e) Round 0 empty slice screening and gallbladder refinement.

Key Results

  • Superior Accuracy: NativeSeg reached 86.81% adult DSC and 87.04% zero-shot pediatric DSC.
  • Massive Gains: It improved Round 0 performance by +19.81 points over discrete codebook baselines and +7.04 points over 1D [SEG] broadcast methods on the same backbone.
  • Zero-Shot Specialist: Despite being trained only on adults, it outperformed several multi-dataset foundation models (like BiomedParse and 3D SegVol) on pediatric data.
  • Safe Refinement: When used to refine 3D VoxTell, it raised the DSC to 89.85%, whereas other common refiners (like LRN) actually degraded the base masks.

Limitations and Future Work

While powerful, NativeSeg has a few hurdles:

  • Memory Footprint: Running a 27B parameter backbone requires significant VRAM (~54GB). The authors suggest low-bit quantization as a next step to make it more accessible.
  • Anatomical Scope: The current study focused on seven abdominal organs. Future work is needed to extend this to thoracic, vascular, and musculoskeletal structures, as well as specific lesion targets.
  • Temporal Context: While the 5-slice window provides some 3D context, the model does not yet have a full-volume 3D token memory, which could further improve consistency across very large volumes.

Summary updated

Abstract

Text prompted 3D computed tomography (CT) segmentation requires both full slice context and fine boundary precision across volumes where over $80\%$ of organ prompted slices lack the queried target. Existing medical vision language models (VLMs) are largely non native segmenters: they delegate pixel delineation outside the VLM by compressing reasoning into a 1D [SEG] vector for an external Segment Anything Model (SAM) decoder, quantizing masks into lossy discrete codebooks, or issuing box prompts to stateless external SAM tools without empty slice screening or spatial state memory. We introduce NativeSeg, a native vision language segmentation framework on Low Rank Adaptation (LoRA) adapted Qwen3.8-27B that performs dense mask decoding, volumetric triage, spatial error self diagnosis, and multiround local zoom directly over the VLM's 2D spatial visual tokens without external SAM decoders. NativeSeg decodes confidence maps from upper layer 2D spatial tokens fused with a five slice U-Net pathway, halts at zero zooms on $>91\%$ of standalone evaluations via a native presence gate (<pres>), localizes uncertain boundaries via a 20 cell multiscale error gate (<err>), and accepts zoomed updates only when predicted error decreases. Trained solely on adult CT, NativeSeg improves Round $0$ DSC by $+19.81$ and $+7.04$ points over discrete codebook and 1D [SEG] readouts on the identical backbone, reaches $86.81\%$ adult and $87.04\%$ zero shot pediatric DSC, and safely refines 2D nnU-Net ($89.52\%$ adult; $86.86\%$ pediatric), 3D SegVol ($86.81\%$), and 3D VoxTell ($89.85\%$).
BibTeX
@misc{ma2026nativeseg,
  author = {Chengqian Ma and Gaole Dai and Zijie Meng and Weixuan Jin and Yijie Du and Yuexiao Ma and Xiawu Zheng and Fei Chao},
  title = {{NativeSeg: Native Vision Language Segmentation of 3D Computed Tomography via Spatial Visual Tokens and Verified Local Zoom}},
  year = {2026},
  eprint = {2611.24443},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24443},
}
Figure