AG-JEPA: Anatomically Guided Joint Embedding Predictive Architecture for Neurodegenerative Representation Learning

Azad Singh, Ayantika Das, Amog Rao, Aiman Farooq, Anirban Porya, Ananya Shukla, Sowbhagya Suresh, Srinivasan Iyengar, Mohit Jain, Mercy Prasanna RanjitarXiv 2026 · arXiv

TL;DR

Researchers from Microsoft Research India have developed AG-JEPA, a new self-supervised pretraining framework that "teaches" AI models to focus on specific brain regions critical to Alzheimer’s disease. By grounding representations in neuroanatomy and modeling disease progression as simple displacements in a latent space, they achieved an 8.4% boost in early-stage diagnosis and state-of-the-art results in predicting future brain atrophy without needing to generate synthetic images.

What Problem Does This Paper Address?

Detecting Alzheimer’s Disease (AD) in its earliest stages—specifically the transition from being "cognitively normal" (CN) to having "mild cognitive impairment" (MCI)—is notoriously difficult. At this stage, brain changes are subtle and localized to tiny structures like the hippocampus.

Current "foundation models" for neuroimaging often use a Joint Embedding Predictive Architecture (JEPA). Unlike generative models that try to reconstruct every pixel, JEPA models learn by predicting the latent representations of masked image patches. While powerful, these models are "anatomically blind"—they treat all brain patches equally. This lack of spatial awareness means they often miss the faint, region-specific signals that are the "smoking guns" of early neurodegeneration.

Methods

The authors introduce two main components: AG-JEPA for representation learning and Prog-AG-JEPA for longitudinal modeling.

1. AG-JEPA (Anatomically Guided JEPA)

The core innovation is the introduction of anatomical tokens. In addition to standard image patches, the model is given learnable tokens representing five key bilateral regions: the hippocampus, entorhinal cortex, amygdala, temporal horn, and lateral ventricle.

Overview of the AG-JEPA architecture showing how anatomical tokens interact with patch tokens.

To ensure these tokens actually learn something useful, the authors impose three constraints during pretraining:

  • Cross-Stream Prediction: Anatomical tokens must predict masked image patches, and visible patches must predict anatomical tokens. This forces the tokens to capture the "essence" of their respective regions.
  • Region Alignment: Each token is explicitly aligned with a pooled representation of its corresponding anatomical region (using a "coverage matrix" derived from standard segmentation tools).
  • Attention Regularization: The model is penalized if an anatomical token "looks" at the wrong part of the brain.

2. Prog-AG-JEPA

Building on this anatomically organized latent space, the authors propose a lightweight head to model disease progression. Instead of the computationally expensive and "hallucination-prone" task of synthesizing a future MRI scan, Prog-AG-JEPA predicts the latent drift—how the embeddings will change over time—conditioned on the patient's age, current brain volumes, and the time interval.

The Prog-AG-JEPA head predicts future latent states as displacements from the current state.

Datasets

The models were primarily trained and evaluated on the ADNI (Alzheimer’s Disease Neuroimaging Initiative) dataset. To test real-world robustness (cross-cohort generalization), they evaluated the models on three additional datasets without any further fine-tuning:

  • MR-RATE: A large-scale clinical dataset.
  • AIBL: Australian Imaging, Biomarkers and Lifestyle study.
  • MIRIAD: Minimal Interval Resonance Imaging in AD.

Key Insights and Ideas

  • Anatomy as a Prior: Explicitly telling the model where to look for disease signals is significantly more effective than letting it learn from scratch. This is especially true for early-stage detection where the signal-to-noise ratio is low.
  • Interpretability by Design: Because the anatomical tokens are tied to specific regions, we can visualize their attention maps. Unlike baseline models that show diffuse attention, AG-JEPA tokens focus precisely on their assigned structures.
  • Latent Space Trajectories: Modeling disease as a "displacement" in a frozen latent space is a powerful alternative to generative modeling. It captures clinical change (like cognitive score decline) more accurately and efficiently.
  • Separability: As shown in the t-SNE plots below, AG-JEPA’s anatomical tokens form much tighter, more distinct clusters than standard patch-based embeddings, indicating they have captured unique regional characteristics.
t-SNE visualization showing that AG-JEPA's anatomical tokens form distinct, separable clusters compared to baselines.

Key Results

  • Superior Diagnosis: AG-JEPA achieved an 8.4 percentage point AUROC gain over the strongest baseline (Neuro-JEPA) for the difficult CN vs. MCI classification task.
  • Robust Generalization: On the out-of-distribution MR-RATE dataset, it outperformed baselines by 7.1 percentage points in CN/AD classification.
  • Accurate Progression: Prog-AG-JEPA outperformed state-of-the-art generative models (like BrLP and AG-LDM) in predicting future diagnosis and regional volume changes. It produced volume change maps that closely matched the ground truth patterns of atrophy.
Comparison of predicted vs. ground-truth regional volume changes, showing AG-JEPA's superior accuracy.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Fixed Regions: The current model only uses five predefined regions. While these are the most relevant for AD, expanding to a whole-brain parcellation could capture a wider range of neurodegenerative conditions.
  • Code Availability: At the time of the paper's release, the code has not been made public.
  • Future Directions: The team suggests that future work could involve "learned" anatomical priors rather than fixed ones, potentially discovering new regions of interest for different diseases.

Summary updated

Abstract

Brain MRI can reveal early signs of Alzheimers disease (AD) before measurable cognitive decline becomes apparent. However, during the transition from cognitively normal (CN) to mild cognitive impairment (MCI), the disease-related signals are often subtle and confined to only a few brain regions. Existing Joint Embedding Predictive Architecture (JEPA) models learn general structural representations but do not explicitly capture the anatomical regions affected in AD. We propose Anatomically Guided JEPA (AG-JEPA), which introduces anatomical tokens that interact with patch tokens during pretraining to learn region-specific representations from brain regions relevant to neurodegeneration. Building on this representation space, we further propose Prog-AG-JEPA, which models disease progression trajectory as a conditioned latent displacement within this space. We evaluate AG-JEPA across five downstream tasks spanning Alzheimer's disease diagnosis and progression prediction. AG-JEPA improves diagnosis and MCI prognosis, achieving an 8.4 percentage point AUROC gain over the strongest baseline on CN/MCI classification and a 7.1 percentage point gain on out-of-distribution CN/AD classification on the MR-RATE dataset, highlighting its effectiveness across both early-stage detection and cross-cohort generalization. For disease progression modeling, Prog-AG-JEPA consistently outperforms all baselines on future diagnosis and brain volume change prediction, demonstrating the value of anatomically grounded neurodegeneration representations for diagnosis and prognosis.
BibTeX
@misc{singh2026ag,
  author = {Azad Singh and Ayantika Das and Amog Rao and Aiman Farooq and Anirban Porya and Ananya Shukla and Sowbhagya Suresh and Srinivasan Iyengar and Mohit Jain and Mercy Prasanna Ranjit},
  title = {{AG-JEPA: Anatomically Guided Joint Embedding Predictive Architecture for Neurodegenerative Representation Learning}},
  year = {2026},
  eprint = {2611.18097},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18097},
}
Figure