VOILA: Complexity-Aware Universal Segmentation of CT images by Voxel Interacting with Language

Zhenglin Wan, Gao Yu, Wenjie Pang, Da‐Wei DingarXiv 2025 · DOI · arXiv

Abstract

Satisfactory progress has been achieved recently in universal segmentation of CT images. Following the success of vision-language methods, there is a growing trend towards utilizing text prompts and contrastive learning to develop universal segmentation models. However, there exists a significant imbalance in information density between 3D images and text prompts. Moreover, the standard fully connected layer segmentation approach faces significant challenges in handling multiple classes and exhibits poor generalizability. To address these challenges, we propose the VOxel Interacting with LAnguage method (VOILA) for universal CT image segmentation. Initially, we align voxels and language into a shared representation space and classify voxels on the basis of cosine similarity. Subsequently, we develop the Voxel-Language Interaction framework to mitigate the impact of class imbalance caused by foreground-background discrepancies and variations in target volumes. Furthermore, a Complexity-Aware Sampling method is proposed to focus on region hard to segment, achieved by generating pseudo-heatmaps from a trainable Gaussian mixture distribution. Our results indicate the proposed VOILA is capable to achieve improved performance with reduced parameters and computational cost during training. Furthermore, it demonstrates significant generalizability across diverse datasets without additional fine-tuning.

1. Introduction

Universal medical image segmentation—training a single model to identify any organ across various datasets—is a major goal in clinical AI. However, three main hurdles persist in current approaches. First, the "Fixed Head" problem: most models use a standard fully connected (FC) layer for classification, which is dataset-specific and scales poorly as the number of classes increases. Second, there is a massive information imbalance between a 3D CT volume (millions of voxels) and a simple text prompt (a few words). Third, extreme class imbalance between background voxels and foreground organs often leads models to ignore smaller or more complex structures.

To address these challenges, we introduce VOILA (VOxel Interacting with LAnguage), a lightweight, universal framework for 3D CT image segmentation. VOILA replaces traditional classification layers with voxel-to-text contrastive learning and utilizes a "Complexity-Aware Sampling" (CAS) module to focus training on difficult regions like organ boundaries.

2. Methodology

The VOILA architecture is designed to bridge the gap between high-density volumetric data and low-density linguistic prompts through a shared representation space.

Overall Workflow of VOILA, including the Voxel-Language Interaction and Complexity-Aware Sampling modules.

2.1 Voxel-Language Interaction

Instead of an FC layer, VOILA maps every voxel and every class-specific text prompt into a shared representation space. Classification is performed by calculating the cosine similarity between a voxel’s feature and the text tokens. This makes the model "label-agnostic," allowing for the addition of new classes simply by providing new text prompts rather than retraining a classification head. To provide spatial context, the authors enhanced standard templates by adding neighboring information (e.g., "This region represents the {organ}. Neighboring structures include {STR1}, {STR2}..."), encouraging the model to learn spatial topology.

2.2 Complexity-Aware Sampling (CAS)

The CAS module is a self-supervised component using a Conditional Variational Auto-Encoder (CVAE). It generates "pseudo-heatmaps" that identify regions with high segmentation difficulty, such as organ boundaries or small vessels. The model then selectively samples these "hard" voxels for training, drastically reducing computational waste on easy background areas and acting as an automated curriculum that focuses on regions that are difficult to segment.

2.3 Voxel-wise F1 Loss

Since the model treats voxels as independent entities for contrastive learning, the standard Dice loss was adapted into a voxel-wise F1 loss. This adaptation better handles the inherent class imbalance in CT scans and foreground-background discrepancies.

3. Experimental Results

The model was evaluated on 7 public CT datasets, including TotalSegmentator v2 (117 foreground classes), WORD, AMOS, and BTCV.

3.1 Parameter Efficiency and Performance

VOILA is remarkably efficient, using only 6.44 million trainable parameters, compared to 149.46M for nnFormer and 42.97M for UNETR++. Despite this reduction, VOILA achieved state-of-the-art Dice scores on large-scale datasets like Ts-v2 (92.1) and WORD (83.0).

Qualitative comparison of segmentation results between Groundtruth, nnUNet, Unetr++, and VOILA across different anatomical regions.

3.2 Zero-Shot Generalization

When trained on Ts-v2 and tested on other datasets without fine-tuning, VOILA significantly outperformed traditional supervised models. For example, on the BTCV dataset, it improved the Normalized Surface Dice (NSD) by 9.0% compared to a model trained specifically on that data. This demonstrates the robust generalizability of the voxel-language alignment approach across diverse anatomical structures.

4. Conclusion

VOILA offers a scalable and efficient solution for universal CT segmentation. By treating segmentation as a similarity matching task and focusing on complex regions through CAS, it achieves superior performance and generalization with a fraction of the parameters required by modern Transformers. While it shows slightly lower performance on some single-organ tasks where heavy architectures may benefit from redundancy, its efficiency and zero-shot capabilities make it a powerful tool for large-scale clinical applications. Future work will focus on fine-tuning the text encoder for medical terminology and scaling the framework to full-resolution whole-body scans.

Summary updated

Abstract

Satisfactory progress has been achieved recently in universal segmentation of CT images. Following the success of vision-language methods, there is a growing trend towards utilizing text prompts and contrastive learning to develop universal segmentation models. However, there exists a significant imbalance in information density between 3D images and text prompts. Moreover, the standard fully connected layer segmentation approach faces significant challenges in handling multiple classes and exhibits poor generalizability. To address these challenges, we propose the VOxel Interacting with LAnguage method (VOILA) for universal CT image segmentation. Initially, we align voxels and language into a shared representation space and classify voxels on the basis of cosine similarity. Subsequently, we develop the Voxel-Language Interaction framework to mitigate the impact of class imbalance caused by foreground-background discrepancies and variations in target volumes. Furthermore, a Complexity-Aware Sampling method is proposed to focus on region hard to segment, achieved by generating pseudo-heatmaps from a trainable Gaussian mixture distribution. Our results indicate the proposed VOILA is capable to achieve improved performance with reduced parameters and computational cost during training. Furthermore, it demonstrates significant generalizability across diverse datasets without additional fine-tuning.
BibTeX
@misc{wan2025voila,
  author = {Zhenglin Wan and Gao Yu and Wenjie Pang and Da‐Wei Ding},
  title = {{VOILA: Complexity-Aware Universal Segmentation of CT images by Voxel Interacting with Language}},
  year = {2025},
  eprint = {2501.03482},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.03482},
}
Figure