Mask-Aware Execution for Efficient JEPA Training

Md Musfiqur Rahman Sanim, Zhihao Shu, Bahram Afsharmanesh, Amirali Mirian, Wei Niu, Gagan AgrawalarXiv 2026 · DOI · arXiv

TL;DR

Joint Embedding Predictive Architectures (JEPAs) are rapidly becoming the foundation for advanced world models, but training them is highly inefficient due to massive computational redundancies across multiple overlapping masks. This paper introduces M-JEPA, a mask-aware execution framework that restructures the JEPA training pipeline to share common computations and execute sparsely. Without altering the underlying machine learning objective or training semantics, M-JEPA delivers up to a 1.7× end-to-end training speedup and a 4.75× speedup for patch embedding, making large-scale world model training significantly faster and cheaper.

What Problem Does This Paper Address?

Self-supervised learning via JEPAs works by predicting the latent (hidden) representations of masked parts of an input from the unmasked context parts. To maximize sample efficiency, standard JEPA training pairs each input with multiple overlapping mask configurations (typically 2 to 10 masks per training step).

However, current deep learning frameworks execute each mask branch completely independently. This naive multi-mask execution introduces three severe system-level bottlenecks on GPUs:

  1. Redundant Computation: The same input is patchified and run through early encoder layers multiple times, even though up to 40% of the tokens are identical across different masks.
  2. Memory-Bound Token Routing: Extracting specific token subsets for each mask requires frequent gather and scatter operations. These operations access non-contiguous memory locations, destroying cache locality and bottlenecking global memory bandwidth.
  3. Dense Target Encoder Execution: The target encoder (which generates the ground-truth latent targets) is typically run densely over the entire token grid, wasting FLOPs on tokens that never actually contribute to the loss function.

Methods

Instead of changing what the model computes (the loss objective), M-JEPA changes how it executes on the hardware. The authors introduce a mask-aware training architecture built on several novel components:

  • Shared Patch Embedding & Context Encoder: Patch embedding is executed exactly once per input clip. M-JEPA factors out mask-independent layers, running them once, and defers mask-specific token selection to later stages.
  • Fused Operators for Routing and Backpropagation: The authors designed custom fused kernels that build a unified routing map in a single pass. During backpropagation, a fused slicing operator aggregates gradient updates from all mask branches into a single coalesced scatter operation, drastically reducing device-to-device memory traffic.
  • Sparse Target Encoder Execution: Instead of running the target encoder densely, M-JEPA computes the mathematical union of all target tokens across all masks. The target encoder is then executed sparsely only on this compact union set.
Figure 5: Overview of optimized mask-aware execution in M- JEPA. Shared computation in the context encoder is executed once per input, mask-dependent work is handled through lightweight routing, and the target encoder operates sparsely on the union of target tokens before routing representations to the predictor.

Datasets

The authors evaluated M-JEPA across five distinct JEPA workloads spanning various modalities to demonstrate its generality:

  • V-JEPA: Video modality (used as the primary, most demanding benchmark due to its heavy spatiotemporal tokenization).
  • EEG-JEPA: Brain signal analysis (Video-to-EEG).
  • T-JEPA: Tabular data.
  • TS-JEPA: Time-series data.
  • Brain-JEPA: Brain dynamics foundation modeling.

Key Insights and Ideas

  • Execution Restructuring over Algorithmic Tweaks: A common trend in ML efficiency research is modifying the objective function or freezing layers to save compute. M-JEPA proves that massive efficiency gains are left on the table simply due to poor execution dataflow in multi-branch architectures.
  • The "Union" Efficiency Window: For the target encoder, running a single sparse pass over the union of tokens is significantly faster than running multiple independent masked passes or one full dense pass.
  • Compiler Limitations: Standard graph compilers like torch.compile optimize local subgraphs but fail to hoist mask-independent computations across entirely separate execution branches. M-JEPA fills this gap by explicitly restructuring the macro-dataflow.

Key Results

  • End-to-End Speedups: Evaluated on NVIDIA A100 GPUs, M-JEPA achieved up to a 1.7× end-to-end training speedup for V-JEPA as the mask count scaled to 10. Shared execution alone contributed to ~80% of this latency reduction.
  • High-Sparsity Acceleration: For single-mask configurations with large input resolutions (1024×1024), the custom masked patch embedding delivered a 4.75× speedup by completely skipping unused patches.
  • Multi-GPU Scalability: The performance benefits successfully carried over to distributed settings, maintaining a ~1.5× speedup in 2-GPU and 4-GPU parallel training configurations.
  • Perfect Mathematical Fidelity: Crucially, because the optimizations do not alter the mathematical semantics of the model, the training loss convergence curves for M-JEPA and the baseline were completely identical.

Limitations and Future Work

  • Activation Memory Overhead: Because intermediate tensors must be held in memory until all mask branches finish processing them, M-JEPA increases peak GPU memory usage by up to 10% in 10-mask configurations.
  • Sparsity Bounds: If a multi-mask configuration is so dense that the union of target tokens covers nearly 100% of the input space, the benefits of sparse target encoding diminish toward the dense baseline.

Future Directions: The authors plan to scale these fused mask routing and sparse execution techniques to multi-node, low-bit (FP8/INT8) training environments, as well as extending mask-aware execution to accelerate JEPA-based world-model inference.

Summary updated

Abstract

Joint Embedding Predictive Architectures (JEPAs) are becoming a core representation-learning primitive and a building block for latent world models across vision, video, audio, brain dynamics, and time series. Despite (potential of) wide deployment, current JEPA training pipelines are inefficient: each input is executed through multiple mask-specific branches, with redundant target-side work, and memory-bound token routing. These costs grow with the number of masks and limit GPU efficiency. We present M-JEPA, a mask-aware execution architecture that restructures JEPA training without changing the learning objective. M-JEPA separates mask-independent computation from mask-dependent routing, enabling shared context encoder execution, fused token routing and slicing with backward support, sparse target encoder execution over the union of target tokens, and masked patch embedding for sparse inputs. The resulting pipeline preserves training semantics while reducing computation, memory traffic, and synchronization overhead. We implement M-JEPA for five JEPA variants and evaluate it on NVIDIA A100 GPUs. Compared against the state-of-the-art baselines, M-JEPA achieves up to 1.7x end-to-end training speedup for 2-10 masks. Separately, with masked patch embedding, 4.75x patch-embedding speedup at high sparsity. These results show that execution restructuring, rather than changes to the JEPA objective, is a key lever for efficient JEPA training.
BibTeX
@misc{sanim2026mask,
  author = {Md Musfiqur Rahman Sanim and Zhihao Shu and Bahram Afsharmanesh and Amirali Mirian and Wei Niu and Gagan Agrawal},
  title = {{Mask-Aware Execution for Efficient JEPA Training}},
  year = {2026},
  eprint = {2609.22674},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.22674},
}
Figure