CascadeJEPA: Enhancing 3D Geometric Representations via Spatially Coupled Cascaded Prediction

Kailai Zhou, Ziqi Chen, Xiaoning Xu, Yuefeng Wu, Zixuan Fu, Zijun Lin, Zhihao Li, Youwei Pang, Xiaoqi Zhao, Xun Cao, Bihan WenarXiv 2026 · arXiv

TL;DR

CascadeJEPA fixes a major blind spot in predictive visual models: the tendency to lose 3D geometric detail as they become more semantically "intelligent." By introducing a two-stage cascaded prediction framework with full-grid spatial context, it significantly improves performance on tasks like depth estimation and feature matching while maintaining top-tier semantic recognition.

What Problem Does This Paper Address?

Joint-Embedding Predictive Architectures (JEPAs), like the popular I-JEPA, learn by predicting the latent representations of masked image patches from visible ones. While this is a powerful way to learn "what" is in an image (semantics), the authors discovered a troubling trend: as these models train longer and get better at classification, their understanding of "where" things are in 3D space (geometry) actually degrades.

The authors pinpoint two main culprits for this "geometric decay":

  1. Incomplete Spatial Context: Standard I-JEPA only processes visible patches, leaving the masked areas completely absent from the encoder's grid. This prevents the model from learning fine-grained spatial relationships across the whole image.
  2. Single-Stage Abstraction: Predicting highly abstract, final-layer features in one go encourages the model to favor predictable semantic patterns over the messy, detailed geometric variations needed for tasks like depth estimation.
Analysis of I-JEPA showing the discrepancy between semantic and geometric learning over time.

Methods

To bridge this gap, the authors propose CascadeJEPA, which modifies the I-JEPA architecture with three key innovations:

  • Cascaded Architecture: Instead of a single prediction, the model uses two stages. The first stage learns a spatially structured "map" of the image, which then guides the second stage's latent prediction via a cross-attention module.
  • Full-grid Context Encoding: In the first stage, the model doesn't just discard masked patches. Instead, it keeps them as "zero tokens" in the grid. This allows the Transformer encoder to model interactions across the entire spatial layout, even for parts it can't "see" yet.
  • Visibility-Enhanced Spatial Coupling: The first stage is given a higher proportion of visible patches (reducing masked blocks from 4 to 2). This provides a richer "spatial anchor" to support the more abstract predictions in the second stage.
Comparison of the original I-JEPA (left) and the proposed CascadeJEPA (right) architectures.

Datasets

The model was pretrained on ImageNet-1K. For evaluation, the authors used a diverse battery of benchmarks:

  • Geometric Tasks: NAVI (object-centric matching and depth), ScanNet (scene-level matching), and NYUv2 (surface normals and depth estimation).
  • Semantic Tasks: SPair-71k (semantic correspondence) and ImageNet-1K (linear probing and fine-tuning).
  • Downstream Applications: DINO-Foresight for future feature forecasting and DINO-WM for world model action planning.

Key Insights and Ideas

  • The "Geometric Over-compression" Phenomenon: Through subspace analysis, the authors found that as I-JEPA trains, depth-related information gets squashed into fewer and fewer dimensions. While the model becomes "efficient," it loses the nuance required for accurate 3D reconstruction.
  • Intermediate Layers are Geometric Goldmines: In standard JEPAs, geometric correspondence peaks in middle layers and plummets in the deepest layers. CascadeJEPA's cross-attention mechanism helps "rescue" this spatial information, carrying it through to the final representation.
  • Context is King for Geometry: Simply allowing the encoder to see the positions of masked patches (the full-grid encoding) significantly boosts the model's ability to handle spatial relationships, even before adding the cascaded prediction.
Figure 2. (a) Layer-wise feature matching on NAVI shows that geometric correspondence in I-JEPA peaks at intermediate layers and degrades substantially in deeper layers. (b) Subspace inter- vention on depth estimation reveals increasing concentration of depth-relevant information as pretraining progresses.

Key Results

CascadeJEPA achieves a "best of both worlds" result, outperforming I-JEPA and other prediction-driven models across the board:

  • Feature Matching: On ScanNet, it achieved a 26.11% Recall@5px, a massive jump from I-JEPA's 1.52%.
  • 3D Structure: On NYUv2 surface normal estimation, it improved the strictest accuracy metric (11.25°) by 7.28 percentage points over I-JEPA.
  • Semantics: It reached 83.50% Top-1 accuracy on ImageNet-1K with only 300 epochs of pretraining, surpassing I-JEPA's 600-epoch performance (82.52%).
  • World Modeling: In robotic planning simulations (PointMaze, Wall), CascadeJEPA-based agents navigated obstacles more effectively, increasing the average success rate from 63.1% to 74.0%.

Limitations and Future Work

  • Computational Overhead: The two-stage prediction and cross-attention modules add extra complexity and training time compared to the original I-JEPA.
  • Hyperparameter Sensitivity: The performance is sensitive to the masking ratio and the specific placement of the cross-attention layers.
  • Future Directions: The authors suggest extending this cascaded approach to video (V-JEPA) to see if it can similarly enhance spatio-temporal geometric representations for more complex world models.

Summary updated

Abstract

Joint-Embedding Predictive Architectures (JEPAs) have emerged as a promising paradigm for predictive visual representation learning. However, our analysis of I-JEPA reveals a growing discrepancy between semantic and geometric learning: improved semantic recognition during pretraining is accompanied by degraded 3D geometric representations. We hypothesize that this partly stems from predicting highly abstract latent representations with incomplete spatial context, favoring semantic regularities over fine-grained geometric information. To address this issue, we introduce CascadeJEPA, a spatially coupled cascaded prediction framework that promotes richer patch-level interactions through full-grid context encoding and cross-attention between prediction stages, while largely preserving the original I-JEPA architecture. Extensive experiments demonstrate substantial improvements over I-JEPA across multiple 3D geometric tasks while maintaining strong semantic recognition performance. Furthermore, its advantages extend to future feature forecasting and world model planning, highlighting the potential of enhanced geometric representations to support predictive world modeling.
BibTeX
@misc{zhou2026cascadejepa,
  author = {Kailai Zhou and Ziqi Chen and Xiaoning Xu and Yuefeng Wu and Zixuan Fu and Zijun Lin and Zhihao Li and Youwei Pang and Xiaoqi Zhao and Xun Cao and Bihan Wen},
  title = {{CascadeJEPA: Enhancing 3D Geometric Representations via Spatially Coupled Cascaded Prediction}},
  year = {2026},
  eprint = {2611.22686},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22686},
}
Figure