Cracks in Embeddings: Robustness and Stability of JEPA-Based Chest X-Ray Representations

Zahid Mansur, Kalpana Yadav, Deepak MishraarXiv 2026 · arXiv

TL;DR

While Joint Embedding Predictive Architecture (JEPA) models are gaining traction in medical AI, they are surprisingly fragile to common image corruptions and adversarial attacks. This paper demonstrates that while these models "crack" under pressure, their robustness can be significantly restored by fine-tuning a lightweight predictor head on perturbed data, whereas architectural fixes like discrete codebooks fail to provide a shortcut to stability.

What Problem Does This Paper Address?

Deep learning in medicine faces a "label desert"—expert annotations are expensive and rare. Self-supervised learning (SSL) offers a solution by learning from unlabeled data. A recent star in the SSL world is JEPA, which learns by predicting the representation of missing image parts rather than raw pixels. This encourages the model to capture high-level anatomical structures rather than low-level noise.

However, clinical reality is messy. X-ray machines vary by hospital, settings change, and sensors introduce noise. If a model is trained on "clean" data but encounters a slightly darker or noisier image in the clinic, will it fail silently? The authors investigate the robustness of JEPA-based chest X-ray (CXR) encoders against both realistic image corruptions and malicious black-box adversarial attacks.

Methods

The researchers evaluated four JEPA-style encoders: CheXWorld, RadJEPA, I-JEPA (trained on natural images), and a custom-trained X-WIN.

The core methodology involves keeping the massive encoder frozen (simulating a scenario where a researcher uses a pre-trained model they cannot afford to retrain) and attaching a small, trainable "predictor" (a 3-layer residual MLP) and a linear classifier on top.

Figure 1: (a) Training: each frozen encoder trains its own predictor/classifier pair on clean images, not shared across models. (b) Test: the held-out set is perturbed, passed through the same frozen encoders, and evaluated by two independently trained predictors, codebook (clean images only) and fine-tuned (clean and perturbed images).

They tested two strategies to improve robustness:

  1. Adversarial Fine-Tuning: Training the small predictor head on a mix of clean and perturbed images. They used a loss function that specifically penalizes the model if the embedding of a noisy image drifts away from the embedding of its clean counterpart.
  2. Vector-Quantized (VQ) Codebook: Replacing a middle layer of the predictor with a discrete "lookup table" of 128 learned vectors. The hypothesis was that a discrete bottleneck might "snap" noisy inputs back to a clean, known representation.

Datasets

  • NIH ChestX-ray14: The primary dataset used for training the predictors and evaluation (78,468 training images, 3,000 test images).
  • LIDC-IDRI CT: Used to pre-train the X-WIN encoder.
  • ImageNet: The original pre-training source for the I-JEPA encoder.

Key Insights and Ideas

  • The "Generalist" Fragility: I-JEPA, which was trained on general natural images (ImageNet), achieved the highest accuracy on clean data and was the most label-efficient. However, it was also the most fragile, showing massive performance drops when faced with noise.
  • Embeddings Crack Before Predictions Fail: Even when the classification accuracy (AUROC) stayed relatively stable, the underlying embeddings often collapsed. For instance, the Cosine Similarity between clean and noisy embeddings dropped as low as 0.33. This suggests models might be "right for the wrong reasons" when faced with distortion.
  • Architecture vs. Data: A major "aha" moment was that the VQ Codebook actually made things worse (dropping AUROC by up to 24% on noisy data). This contradicts some prior work in natural images and suggests that for medical imaging, a discrete bottleneck alone isn't a substitute for seeing perturbed data during training.
  • Lightweight Fixes Work: You don't need to retrain a 100M+ parameter encoder to get robustness. Fine-tuning the tiny (3-layer) predictor head was enough to bring embedding consistency back above 0.97.

Key Results

The authors measured damage using Macro AUROC and Embedding Cosine Similarity.

  • Baseline Fragility: Under the strongest perturbations, CheXWorld degraded by 6.90%, X-WIN by 3.17%, RadJEPA by 4.76%, and I-JEPA by a staggering 10.77%.
  • Adversarial Fine-Tuning: This method was highly effective. For I-JEPA, it provided a +0.10 AUROC gain on perturbed data compared to the baseline.
  • Label Efficiency: In the "Few-Sample" test, I-JEPA proved superior, maintaining a 0.65 AUROC with only 1% of training labels, outperforming encoders trained specifically on medical data.
Comparison of AUROC across different severity levels for various perturbations.

Limitations and Future Work

  • Black-Box Only: The study focused on black-box attacks (SPSA and Boundary Attack). While these are more realistic for deployed systems, white-box attacks (where the attacker knows the model gradients) might reveal even deeper vulnerabilities.
  • Codebook Scale: The VQ codebook used only 128 entries. The authors suggest that a much larger codebook or one trained jointly with the encoder might perform better, though this would increase computational costs.
  • Dataset Constraints: Due to hardware limits, X-WIN was trained on a smaller dataset than its original implementation, which may have impacted its baseline performance.

In conclusion, the paper warns that JEPA's high-level feature learning does not automatically grant it "immunity" to image noise. Robustness in medical AI is not a default setting; it must be explicitly measured and trained for.

Summary updated

Abstract

Self-supervised Joint Embedding Predictive Architecture (JEPA) models are increasingly used for medical imaging, however there is a limited understanding about robustness of these models toward real-world image changes. We test four JEPA-based chest X-ray encoders, trained on 78,468 NIH ChestX-ray14 images, under six tests: two black-box adversarial attacks (SPSA, Boundary Attack), three image corruptions (Gaussian noise, brightness shift, contrast shift), and a label-scarcity test, measuring embedding change and classification AUROC. These perturbations cause severe, model-dependent damage: encoder-level Cosine Similarity drops as low as 0.33, and AUROC falls by as much as 0.12 in the worst case (I-JEPA under the Boundary Test). We test two ways of training the small predictor on top of each frozen encoder to address this. Adversarial fine-tuning, trained on clean and perturbed images together, raises embedding consistency above 0.97 in nearly every case and improves AUROC, relative to the same perturbed baseline, for most models, with the largest gain of $+0.10$ on I-JEPA. A Vector-Quantized codebook method, trained only on clean images and evaluated on perturbed test images for the first time, instead degrades under distortion for all four models, most noticeably for I-JEPA ($-0.15$ AUROC on average), showing that its discrete bottleneck alone does not provide the robustness that comes from training on perturbed data directly.
BibTeX
@misc{mansur2026cracks,
  author = {Zahid Mansur and Kalpana Yadav and Deepak Mishra},
  title = {{Cracks in Embeddings: Robustness and Stability of JEPA-Based Chest X-Ray Representations}},
  year = {2026},
  eprint = {2611.17905},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17905},
}
Figure