Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomography

Quoc-Huy Trinh, Minh-Van Nguyen, Ulas BagciarXiv 2026 · arXiv

TL;DR

Rad-JEPA 3D is a self-supervised joint-embedding predictive framework designed to learn robust, structurally coherent representations of 3D Computed Tomography (CT) scans. By combining a hybrid Mamba-Transformer encoder with a novel layer-wise orthogonal regularization strategy, it captures critical cross-slice continuity and spatial geometry without wasting capacity on reconstructing pixel-level noise. When integrated into Vision-Language Models (VLMs), it achieves state-of-the-art performance on 3D medical visual question answering and spatial reasoning benchmarks while using significantly fewer parameters than competing models.

What Problem Does This Paper Address?

3D medical imaging modalities like CT and MRI are inherently volumetric, meaning that anatomical structures and lesions span across multiple axial slices. To perform effective diagnostic and spatial reasoning, models must understand both cross-plane context and inter-slice continuity.

While Vision-Language Models (VLMs) have recently excelled at 2D medical tasks, extending them to 3D volumetric reasoning remains a massive challenge. Existing 3D volumetric encoders often fail to preserve coarse spatial and geometric structures. Many rely on masked image modeling (reconstructing raw voxels/pixels), which forces the network to waste capacity on low-level scanner noise and fine textures rather than high-level anatomical semantics. Consequently, when paired with large language models, these encoders produce dense features that lack the structural coherence needed for complex spatial reasoning and organ disentanglement.

Methods

The authors introduce Rad-JEPA 3D, which moves away from pixel-space reconstruction in favor of feature-space prediction using a Joint-Embedding Predictive Architecture (JEPA).

Figure 1: Overall architecture of Rad-JEPA 3D. A CT volume is tokenized into cubes and passed to two weight-sharing en- coders: a student that sees a masked view and an EMA teacher (stop-gradient) that sees the full volume. A predictor aligns student features to the teacher’s at the masked positions via the JEPA loss. Each encoder stacks H-Mamba layers (right), which fuse a Mamba and GQA branch through a per-token router; HSOR regularizes the student’s per-layer hidden states.

Key components of the architecture include:

  • Cube Tokenizer: Instead of processing slices in isolation, the input volume is partitioned into non-overlapping $8\times8\times8$ cubes and embedded into tokens, allowing the model to inherently represent coarse 3D spatial structures.
  • H-Mamba Encoder: To capture both sequential continuity and global context, each encoder layer processes tokens through two parallel branches: a Mamba-2 branch (providing linear-time modeling of long-range inter-slice dependencies) and a Grouped-Query Attention (GQA) branch (capturing all-to-all cross-plane spatial context). A lightweight per-token router dynamically fuses their outputs.
  • Hidden States Orthogonal Regularization (HSOR): To prevent intermediate encoder representations from becoming redundant or losing discriminative power across deep layers, HSOR introduces two penalties:
  • Cross-correlation alignment: Aligns student and teacher hidden states using a Barlow Twins-style loss to eliminate feature redundancy.
  • Weight orthogonality: Applies a soft orthogonality penalty to the encoder's weight matrices to prevent filter collapse.

During self-supervised pretraining, a student network processes a heavily masked volume (75% of tokens removed), and an MLP predictor attempts to predict the latent features of the full volume as represented by an Exponential Moving Average (EMA) teacher network.

Datasets

  • Pretraining: The model was pretrained on approximately 120,000 unique 3D CT volumes from the publicly available M3D-Cap dataset.
  • Instruction Tuning & Evaluation: Post-training for downstream tasks utilized M3D-VQA, which provides roughly 662,000 instruction-response pairs.
  • Spatial Reasoning Evaluation: The model was benchmarked on SpatialMed, a multiple-choice question dataset specifically designed to evaluate directional (DIR), extent/size/shape (EXT), volume-magnitude (VOL), and comparative (COMP) spatial reasoning in 3D medical imaging.

Key Insights and Ideas

  • Latent Prediction > Pixel Reconstruction: Predicting representations in a learned latent space allows the model to focus entirely on high-level anatomical semantics and spatial geometry, ignoring meaningless scanner noise.
  • Spatial Structure Beats Model Scale: Better spatial and geometric organization within a compact 19M-parameter visual encoder can substitute for massive language-model scaling. Rad-JEPA 3D paired with a 3B or 4B LLM outperforms models with twice the parameters.
  • Mamba and Attention are Complementary: Sequential scanning (Mamba) handles the continuous transition between axial slices, while attention (GQA) handles the global cross-plane context. The router effectively exploits both, yielding a significant boost in performance when combined.
  • Aggressive Masking is Mandatory: A high masking ratio (75%) is required to force the model to perform global spatial inference. Lower masking ratios allow the network to cheat by using local interpolation, weakening its grasp of global anatomy.

Key Results

Rad-JEPA 3D achieves state-of-the-art results across multiple benchmarks despite its compact size:

  • Frozen-Encoder Probing: Under a frozen $k$-NN probe, Rad-JEPA 3D significantly outperforms prior self-supervised methods (Swin UNETR, VoCo-B, Models Genesis) across scan plane, contrast phase, and organ classification tasks.
Figure 2: Frozen-encoder k-NN probing accuracy on three CT tasks. Encoders are self-supervised and kept frozen; a k-NN probe classifies their representations, isolating rep- resentation quality from fine-tuning. Tasks are ordered by increasing granularity: scan plane (chance 50.0%), contrast phase (chance 10.0%), and organ (chance 2.9%). RadJepa- Hybrid leads on all three, with the largest margin on the finest-grained task.
  • Closed-Ended VQA: Paired with Qwen3-4B, it achieves a state-of-the-art mean accuracy of 81.66% on the M3D-VQA benchmark, outperforming the 7.6B Med3DVLM (79.75%), with massive gains in abnormality detection (+10.83%).
  • Open-Ended VQA: In free-text answer generation, it achieves 86.16 BLEU-1 and 60.81 METEOR, vastly outperforming Med3DVLM (52.38 BLEU-1, 36.76 METEOR), showing that high-quality visual representations matter immensely when the model cannot rely on multiple-choice answer priors.
  • Spatial Reasoning: It sets a new high score on the SpatialMed benchmark with an average score of 58.16%, driven by a massive +9.47 point margin on volume-magnitude (VOL) reasoning, which requires integrating structure across many axial slices.

Limitations and Future Work

Not explicitly discussed in the paper.

Summary updated

Abstract

Self-supervised pretraining is central to 3D medical image analysis, where unlabeled CT volumes are abundant but expert annotations are scarce. Yet existing volumetric encoders often fail to preserve the coarse spatial and geometric structure that downstream reasoning depends on, limiting their performance on organ disentanglement, abnormality detection, and spatial understanding when paired with language models. We introduce Rad-JEPA 3D, a joint-embedding predictive framework that learns volumetric CT representations by predicting the latent features of a complete scan from a masked view. At its core is a hybrid H-Mamba encoder that fuses a Mamba state-space branch, which models inter-slice continuity through sequential scanning, with a grouped-query attention branch, which captures cross-plane spatial context, combined through a lightweight per-token router. To improve the quality of intermediate representations, we further propose Hidden States Orthogonal Regularization (HSOR), which aligns student-teacher hidden states and reduces feature redundancy throughout the encoder. This layer-wise regularization produces more consistent and discriminative volumetric representations, leading to improved performance on organ recognition and spatial reasoning tasks. Pretrained on approximately 120,000 CT scans, Rad-JEPA 3D attains state-of-the-art results despite its compact size: with only 4.0B total parameters, it achieves competitive results with state-of-the-art on closed-ended VQA and the best average spatial-reasoning score on the Spatial-Med benchmark. Ablation studies confirm that the hybrid block and HSOR contribute complementary gains, and that the induced spatial structure can substitute for raw language-model scale on volumetric reasoning tasks.
BibTeX
@misc{trinh2026rad,
  author = {Quoc-Huy Trinh and Minh-Van Nguyen and Ulas Bagci},
  title = {{Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomography}},
  year = {2026},
  eprint = {2607.26196},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.26196},
}
Figure