VOILA: Complexity-Aware Universal Segmentation of CT Images by Voxel Interacting with Language

Zishuo Wan, Yu Gao, Wanyuan Pang, Dawei DingAAAI 2025 · DOI

Introduction

Medical image segmentation is a critical component of modern diagnostic workflows and treatment planning. While deep learning has revolutionized this field, most current models are "specialists," trained for specific organs or datasets, and often fail to generalize to unseen anatomical structures or different clinical settings.

Universal segmentation models, inspired by vision-language (VL) paradigms like CLIP, aim to overcome these limitations. However, they face two significant challenges: information density mismatch (millions of 3D voxels vs. sparse text prompts) and class imbalance (background voxels dominate, and "easy" central organ voxels often overshadow the "hard" boundary voxels where errors typically occur).

The VOILA Framework

VOILA (VOxel Interacting with LAnguage) addresses these hurdles through a novel architecture that aligns 3D voxels with language tokens in a shared feature space.

The overall workflow of VOILA, detailing the Voxel-Language Interaction and the Complexity-Aware Sampling (CAS) modules.

The framework introduces three primary innovations:

  1. Voxel-Language Interaction: Instead of traditional fixed classification layers, VOILA maps voxels and text prompts into a shared embedding space. Segmentation is achieved by calculating the cosine similarity between voxel features and text tokens, allowing for open-vocabulary capabilities.
  2. Enhanced Text Prompts: To provide richer context, prompts incorporate spatial relationships (e.g., "Neighboring structures include {Stomach}, {Kidney}..."), helping the model learn relative anatomical positioning.
  3. Complexity-Aware Sampling (CAS): This module uses a Conditional Variational Auto-Encoder (CVAE) to generate "pseudo-heatmaps" that identify difficult-to-segment regions, such as organ boundaries. By oversampling these high-complexity voxels during training, the model focuses its learning capacity where errors are most likely to occur.

Experimental Results and Generalization

The researchers evaluated VOILA across seven public CT datasets, including Totalsegmentator v2 (Ts-v2), BTCV, and AMOS. VOILA achieved state-of-the-art performance with a 92.1 Dice score on Ts-v2, outperforming much larger models like nnFormer while using only 6.44 million parameters—a fraction of the 149 million used by traditional transformers.

Visualization of the voxel-language interaction matrix, demonstrating the strong diagonal alignment between specific anatomical text embeddings and voxel tokens.

A key strength of VOILA is its zero-shot generalization. When trained on one dataset and tested on others without fine-tuning, it showed significant improvements over supervised baselines. For instance, on the BTCV dataset, it improved the Normalized Surface Dice (NSD) by +9.0. This robustness stems from the model learning to associate visual features with language concepts rather than static numerical labels.

Conclusion and Future Work

VOILA demonstrates that a voxel-centric, language-aligned approach can achieve high efficiency and superior generalization in medical imaging. By focusing on "hard" voxels through the CAS module, the model achieves faster and more stable convergence.

While VOILA excels in universal tasks, future work will focus on improving performance for specific small organs (like the pancreas) and exploring more descriptive prompts generated by Large Language Models (LLMs). Extending this framework to MRI and PET modalities remains a promising avenue for future research.

Summary updated

BibTeX
@inproceedings{wan2025voila,
  author = {Zishuo Wan and Yu Gao and Wanyuan Pang and Dawei Ding},
  title = {{VOILA: Complexity-Aware Universal Segmentation of CT Images by Voxel Interacting with Language}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32805},
}
Figure