TL;DR
Joint Embedding Predictive Architectures (JEPAs) are rapidly becoming the foundation for advanced world models, but training them is highly inefficient due to massive computational redundancies across multiple overlapping masks. This paper introduces M-JEPA, a mask-aware execution framework that restructures the JEPA training pipeline to share common computations and execute sparsely. Without altering the underlying machine learning objective or training semantics, M-JEPA delivers up to a 1.7× end-to-end training speedup and a 4.75× speedup for patch embedding, making large-scale world model training significantly faster and cheaper.
What Problem Does This Paper Address?
Self-supervised learning via JEPAs works by predicting the latent (hidden) representations of masked parts of an input from the unmasked context parts. To maximize sample efficiency, standard JEPA training pairs each input with multiple overlapping mask configurations (typically 2 to 10 masks per training step).
However, current deep learning frameworks execute each mask branch completely independently. This naive multi-mask execution introduces three severe system-level bottlenecks on GPUs:
- Redundant Computation: The same input is patchified and run through early encoder layers multiple times, even though up to 40% of the tokens are identical across different masks.
- Memory-Bound Token Routing: Extracting specific token subsets for each mask requires frequent
gatherandscatteroperations. These operations access non-contiguous memory locations, destroying cache locality and bottlenecking global memory bandwidth. - Dense Target Encoder Execution: The target encoder (which generates the ground-truth latent targets) is typically run densely over the entire token grid, wasting FLOPs on tokens that never actually contribute to the loss function.
Methods
Instead of changing what the model computes (the loss objective), M-JEPA changes how it executes on the hardware. The authors introduce a mask-aware training architecture built on several novel components:
- Shared Patch Embedding & Context Encoder: Patch embedding is executed exactly once per input clip. M-JEPA factors out mask-independent layers, running them once, and defers mask-specific token selection to later stages.
- Fused Operators for Routing and Backpropagation: The authors designed custom fused kernels that build a unified routing map in a single pass. During backpropagation, a fused slicing operator aggregates gradient updates from all mask branches into a single coalesced
scatteroperation, drastically reducing device-to-device memory traffic. - Sparse Target Encoder Execution: Instead of running the target encoder densely, M-JEPA computes the mathematical union of all target tokens across all masks. The target encoder is then executed sparsely only on this compact union set.
Datasets
The authors evaluated M-JEPA across five distinct JEPA workloads spanning various modalities to demonstrate its generality:
- V-JEPA: Video modality (used as the primary, most demanding benchmark due to its heavy spatiotemporal tokenization).
- EEG-JEPA: Brain signal analysis (Video-to-EEG).
- T-JEPA: Tabular data.
- TS-JEPA: Time-series data.
- Brain-JEPA: Brain dynamics foundation modeling.
Key Insights and Ideas
- Execution Restructuring over Algorithmic Tweaks: A common trend in ML efficiency research is modifying the objective function or freezing layers to save compute. M-JEPA proves that massive efficiency gains are left on the table simply due to poor execution dataflow in multi-branch architectures.
- The "Union" Efficiency Window: For the target encoder, running a single sparse pass over the union of tokens is significantly faster than running multiple independent masked passes or one full dense pass.
- Compiler Limitations: Standard graph compilers like
torch.compileoptimize local subgraphs but fail to hoist mask-independent computations across entirely separate execution branches. M-JEPA fills this gap by explicitly restructuring the macro-dataflow.
Key Results
- End-to-End Speedups: Evaluated on NVIDIA A100 GPUs, M-JEPA achieved up to a 1.7× end-to-end training speedup for V-JEPA as the mask count scaled to 10. Shared execution alone contributed to ~80% of this latency reduction.
- High-Sparsity Acceleration: For single-mask configurations with large input resolutions (1024×1024), the custom masked patch embedding delivered a 4.75× speedup by completely skipping unused patches.
- Multi-GPU Scalability: The performance benefits successfully carried over to distributed settings, maintaining a ~1.5× speedup in 2-GPU and 4-GPU parallel training configurations.
- Perfect Mathematical Fidelity: Crucially, because the optimizations do not alter the mathematical semantics of the model, the training loss convergence curves for M-JEPA and the baseline were completely identical.
Limitations and Future Work
- Activation Memory Overhead: Because intermediate tensors must be held in memory until all mask branches finish processing them, M-JEPA increases peak GPU memory usage by up to 10% in 10-mask configurations.
- Sparsity Bounds: If a multi-mask configuration is so dense that the union of target tokens covers nearly 100% of the input space, the benefits of sparse target encoding diminish toward the dense baseline.
Future Directions: The authors plan to scale these fused mask routing and sparse execution techniques to multi-node, low-bit (FP8/INT8) training environments, as well as extending mask-aware execution to accelerate JEPA-based world-model inference.
Summary updated