Self-supervised learning of imaging and clinical signatures using a multimodal joint-embedding predictive architecture

Thomas Z. Li, Aravind R. Krishnan, Lianrui Zuo, John M. Still, Kim L. Sandler, Fabien Maldonado, Thomas A. Lasko, Bennett A. LandmanarXiv 2025 · DOI · arXiv

Introduction

Predicting lung cancer from pulmonary nodules is a complex task often hampered by the scarcity of labeled multimodal data. While combining imaging (CT scans) with electronic health records (EHRs) should theoretically improve diagnostic accuracy, existing models frequently suffer from overfitting to specific hospital distributions (intra-site heterogeneity). This leads to a failure in generalization when these models are deployed in different clinical settings (inter-site heterogeneity).

This research investigates whether self-supervised learning (SSL) can mitigate these issues. By utilizing a Joint-Embedding Predictive Architecture (JEPA) to pretrain on a massive trove of unlabeled multimodal data, the authors hypothesized that the model would learn robust, transferable features that improve performance upon fine-tuning with limited labeled datasets.

Methodology

The study adapts the Joint-Embedding Predictive Architecture (JEPA) for longitudinal, multimodal medical data.

Data Representation

CT scans were converted into feature vectors using a pre-trained ResNet (Sybil). Longitudinal EHR data, including labs, medications, and diagnoses, were projected into a 2000-dimensional "clinical signature" space using Independent Component Analysis (ICA). These features were then combined into a sequence integrated with time and modality embeddings.

JEPA Pretraining

The architecture utilizes a masking strategy where 15% of the tokens are randomly hidden. A "context encoder" processes the unmasked tokens to predict the latent representations of the masked tokens. Crucially, the model predicts representations rather than raw pixels or text. The target representations are provided by a "target encoder," which is updated via an exponential moving average of the context encoder's weights.

The multimodal JEPA architecture featuring context and target encoders for latent representation prediction.

Experimental Setup and Datasets

The model was evaluated across several cohorts:

  • NoduleVU: A large, unlabeled pretraining dataset from Vanderbilt University Medical Center (VUMC) with over 27,000 CT scans and 24,000 clinical expressions.
  • LiVU-SPN: An internal test cohort used to evaluate intra-site performance.
  • MCL-VUMC: An external test cohort used to evaluate generalization.
  • Synthetic Datasets: To debug unexpected results, the authors created synthetic datasets with simulated nodules and sparse expressions governed by strict causal rules.
Examples of synthetic data and masking used to evaluate the model's ability to infer missing causal variables.

Key Results and Insights

Performance Disparity

The multimodal JEPA model showed promising results on the internal LiVU-SPN cohort, achieving an AUC of 0.905, significantly outperforming the imaging-only baseline (0.733 AUC). However, this success did not translate to the external MCL-VUMC cohort. In the external setting, the simple longitudinal imaging-only model performed best (0.753 AUC), while the multimodal JEPA dropped to 0.724 AUC.

Model Collapse and Overfitting

The authors identified that the multimodal JEPA is highly susceptible to "model collapse," where the model learns trivial, flat representations that minimize loss without capturing meaningful patient features. Furthermore, the increased complexity of multimodal models appears to exacerbate overfitting on the home institution's specific data quirks rather than learning universal indicators of malignancy.

Synthetic Debugging

Controlled experiments on synthetic data revealed that JEPA provided no significant advantage over standard supervised fine-tuning (p=0.635), even in scenarios with extremely scarce labeled data. This suggests that the architecture, in its current form, struggles to learn the underlying causal structure of multimodal clinical data.

Conclusion

While self-supervised learning via JEPA shows potential for internal data enhancement, it currently lacks the stability and generalization required for reliable clinical deployment across diverse institutions. The tendency toward model collapse and the failure to outperform supervised methods in low-data regimes highlight the need for more robust masking strategies and novel loss functions tailored for multimodal EHR and imaging integration.

Summary updated

BibTeX
@misc{li2025self,
  author = {Thomas Z. Li and Aravind R. Krishnan and Lianrui Zuo and John M. Still and Kim L. Sandler and Fabien Maldonado and Thomas A. Lasko and Bennett A. Landman},
  title = {{Self-supervised learning of imaging and clinical signatures using a multimodal joint-embedding predictive architecture}},
  year = {2025},
  eprint = {2509.15470},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.15470},
}
Figure