Abstract
Large-scale, volumetric medical imaging datasets typically aggregate scans from different vendors and devices, resulting in highly variable resolution, slice thicknesses, and numbers of slices per study. Consequently, training representation models usually requires cropping or interpolating along the z-axis to obtain fixed-size blocks, which inevitably causes information loss. We propose a new training approach to overcome this limitation. Instead of absolute position embeddings, we interpret volumes as sequences of 3D chunks and adopt Rotary Position Embeddings, allowing us to treat the z-axis as an unconstrained temporal dimensions. Building on this idea, we introduce a new vision-language model: SigVLP. In SigVLP, we implement Rotary Position Embedding as the positional encoding method, which is applied directly within the attention operation, generating input-conditioned sine and cosine weights on the fly. This design ensures consistent alignment between query and key projections and adapts to any input sizes. To allow for variable input size during training, we sample Computed Tomography volumes in chunks and pair them with localized organ-wise textual observations. Compared to using entire reports for conditioning, chunkwise alignment provides finer-grained supervision, enabling the model to establish stronger correlations between the text and volume representations, thereby improving the precision of text-to-volume alignment. Our models are trained with the Muon optimizer and evaluated on a diverse set of downstream tasks, including zero-shot abnormality and organ classification, segmentation, and retrieval tasks.
Introduction
In medical imaging, particularly Computed Tomography (CT), the variability of data presents a significant challenge for standard vision-language models (VLMs). Unlike 2D natural images, 3D CT scans vary in resolution, slice thickness, and total slice count. Traditional approaches often rely on interpolation or cropping to fit data into fixed-size grids, leading to information loss or loss of clinical context. Furthermore, aligning a full 3D volume with a lengthy radiology report often results in coarse representations where localized abnormalities are overlooked.
Methods: Volume-Adaptive Representation
SigVLP addresses these challenges through three primary innovations designed to make representation learning volume-adaptive:
Rotary Position Embeddings (RoPE)
By adopting RoPE, SigVLP moves away from absolute position embeddings. RoPE encodes relative positions through rotations within the attention mechanism, allowing the model to treat the z-axis as an unconstrained dimension. This enables the model to adapt dynamically to any number of slices without requiring fixed-size inputs or lossy preprocessing.
Chunk-wise Organ Alignment
Instead of global alignment, SigVLP decomposes volumes into 3D chunks of varying lengths (32, 64, or 128 slices). Using an LLM (GPT-5 mini), radiology reports are processed into an "Organ-wise Findings Bank." This allows the model to learn fine-grained correlations between specific anatomical regions and their corresponding textual descriptions, as illustrated in the pipeline below.
Optimization
The framework utilizes Sigmoid loss (inspired by SigLIP) for stable large-scale pre-training and the Muon optimizer to maintain representation stability across variable-length inputs.
Key Results
SigVLP demonstrates superior performance across multiple benchmarks, proving that relative context and fine-grained supervision are critical for 3D medical imaging.
- Retrieval: SigVLP achieved a MeanRank of 8.23 in volume-to-report retrieval, significantly outperforming the previous state-of-the-art, CT-CLIP (MeanRank 26.01).
- Zero-Shot Classification: The model reached 0.80 accuracy in abnormality classification, showing better separation of true positives from hard negatives compared to baselines.
- Anatomical Precision: SigVLP consistently outperforms baselines like DINOv3 and CT-CLIP across various organs. It shows marked improvements in precision for complex structures like the pancreas and kidneys.
- Segmentation: On challenging structures like the aorta, SigVLP improved the Dice score to 0.471, a 1.7x improvement over DINOv3 (0.278), demonstrating its ability to capture fine anatomical details.
Conclusion
SigVLP introduces a robust framework for 3D medical imaging by leveraging relative positional context and fine-grained supervision. By treating the z-axis as a flexible dimension and aligning image chunks with localized text, the model achieves state-of-the-art performance while maintaining computational efficiency. Future work will explore the extension of this architecture to other 3D modalities like MRI and Ultrasound.
Summary updated