TL;DR
Rad-JEPA 3D is a self-supervised joint-embedding predictive framework designed to learn robust, structurally coherent representations of 3D Computed Tomography (CT) scans. By combining a hybrid Mamba-Transformer encoder with a novel layer-wise orthogonal regularization strategy, it captures critical cross-slice continuity and spatial geometry without wasting capacity on reconstructing pixel-level noise. When integrated into Vision-Language Models (VLMs), it achieves state-of-the-art performance on 3D medical visual question answering and spatial reasoning benchmarks while using significantly fewer parameters than competing models.
What Problem Does This Paper Address?
3D medical imaging modalities like CT and MRI are inherently volumetric, meaning that anatomical structures and lesions span across multiple axial slices. To perform effective diagnostic and spatial reasoning, models must understand both cross-plane context and inter-slice continuity.
While Vision-Language Models (VLMs) have recently excelled at 2D medical tasks, extending them to 3D volumetric reasoning remains a massive challenge. Existing 3D volumetric encoders often fail to preserve coarse spatial and geometric structures. Many rely on masked image modeling (reconstructing raw voxels/pixels), which forces the network to waste capacity on low-level scanner noise and fine textures rather than high-level anatomical semantics. Consequently, when paired with large language models, these encoders produce dense features that lack the structural coherence needed for complex spatial reasoning and organ disentanglement.
Methods
The authors introduce Rad-JEPA 3D, which moves away from pixel-space reconstruction in favor of feature-space prediction using a Joint-Embedding Predictive Architecture (JEPA).
Key components of the architecture include:
- Cube Tokenizer: Instead of processing slices in isolation, the input volume is partitioned into non-overlapping $8\times8\times8$ cubes and embedded into tokens, allowing the model to inherently represent coarse 3D spatial structures.
- H-Mamba Encoder: To capture both sequential continuity and global context, each encoder layer processes tokens through two parallel branches: a Mamba-2 branch (providing linear-time modeling of long-range inter-slice dependencies) and a Grouped-Query Attention (GQA) branch (capturing all-to-all cross-plane spatial context). A lightweight per-token router dynamically fuses their outputs.
- Hidden States Orthogonal Regularization (HSOR): To prevent intermediate encoder representations from becoming redundant or losing discriminative power across deep layers, HSOR introduces two penalties:
- Cross-correlation alignment: Aligns student and teacher hidden states using a Barlow Twins-style loss to eliminate feature redundancy.
- Weight orthogonality: Applies a soft orthogonality penalty to the encoder's weight matrices to prevent filter collapse.
During self-supervised pretraining, a student network processes a heavily masked volume (75% of tokens removed), and an MLP predictor attempts to predict the latent features of the full volume as represented by an Exponential Moving Average (EMA) teacher network.
Datasets
- Pretraining: The model was pretrained on approximately 120,000 unique 3D CT volumes from the publicly available M3D-Cap dataset.
- Instruction Tuning & Evaluation: Post-training for downstream tasks utilized M3D-VQA, which provides roughly 662,000 instruction-response pairs.
- Spatial Reasoning Evaluation: The model was benchmarked on SpatialMed, a multiple-choice question dataset specifically designed to evaluate directional (DIR), extent/size/shape (EXT), volume-magnitude (VOL), and comparative (COMP) spatial reasoning in 3D medical imaging.
Key Insights and Ideas
- Latent Prediction > Pixel Reconstruction: Predicting representations in a learned latent space allows the model to focus entirely on high-level anatomical semantics and spatial geometry, ignoring meaningless scanner noise.
- Spatial Structure Beats Model Scale: Better spatial and geometric organization within a compact 19M-parameter visual encoder can substitute for massive language-model scaling. Rad-JEPA 3D paired with a 3B or 4B LLM outperforms models with twice the parameters.
- Mamba and Attention are Complementary: Sequential scanning (Mamba) handles the continuous transition between axial slices, while attention (GQA) handles the global cross-plane context. The router effectively exploits both, yielding a significant boost in performance when combined.
- Aggressive Masking is Mandatory: A high masking ratio (75%) is required to force the model to perform global spatial inference. Lower masking ratios allow the network to cheat by using local interpolation, weakening its grasp of global anatomy.
Key Results
Rad-JEPA 3D achieves state-of-the-art results across multiple benchmarks despite its compact size:
- Frozen-Encoder Probing: Under a frozen $k$-NN probe, Rad-JEPA 3D significantly outperforms prior self-supervised methods (Swin UNETR, VoCo-B, Models Genesis) across scan plane, contrast phase, and organ classification tasks.
- Closed-Ended VQA: Paired with Qwen3-4B, it achieves a state-of-the-art mean accuracy of 81.66% on the M3D-VQA benchmark, outperforming the 7.6B Med3DVLM (79.75%), with massive gains in abnormality detection (+10.83%).
- Open-Ended VQA: In free-text answer generation, it achieves 86.16 BLEU-1 and 60.81 METEOR, vastly outperforming Med3DVLM (52.38 BLEU-1, 36.76 METEOR), showing that high-quality visual representations matter immensely when the model cannot rely on multiple-choice answer priors.
- Spatial Reasoning: It sets a new high score on the SpatialMed benchmark with an average score of 58.16%, driven by a massive +9.47 point margin on volume-magnitude (VOL) reasoning, which requires integrating structure across many axial slices.
Limitations and Future Work
Not explicitly discussed in the paper.
Summary updated