JEPAhead: From Inpainting to Causal Forecasting for Urban Driving

Daniel G. P. Petrini, Davi Pincinato, Fabio Grassiotto, Pedro L. F. Prado, Alexandre F. S. Osorio, Raphael V. Ramos, Felipe PamboukianarXiv 2026 · arXiv

TL;DR

Researchers have successfully adapted the V-JEPA (Visual Joint Embedding Predictive Architecture) from a "fill-in-the-blanks" inpainter to a "what-happens-next" causal forecaster for urban driving. By predicting future states directly in a frozen latent space rather than generating pixels, the model achieves significant gains in identifying future scenarios, particularly for moving vehicles and longer time horizons.

What Problem Does This Paper Address?

Building "world models"—systems that can predict the future state of an environment—is a holy grail for autonomous driving. However, traditional generative models that predict future pixels are computationally expensive and often waste capacity on irrelevant details like the texture of a sidewalk or the flickering of leaves.

The authors of JEPAhead tackle the efficiency and semantic gap by working in latent space. Instead of asking "What will the next image look like?", they ask "What will the high-level features of the next scene be?". Specifically, they investigate whether a model pre-trained for inpainting (predicting missing parts of a video) can be fine-tuned for causal forecasting (predicting the future given the past) while keeping the heavy-lifting encoder frozen.

Methods

The team built upon V-JEPA, a Joint-Embedding Predictive Architecture. Unlike standard autoencoders, JEPA models don't reconstruct pixels during training; they predict the embeddings of masked regions.

Key technical shifts in JEPAhead include:

  • Frozen Encoder: They used a pre-trained V-JEPA1 ViT-L/16 encoder and kept it entirely frozen, ensuring the model relies on existing robust feature representations.
  • Causal Adaptation: While the original V-JEPA was bidirectional (looking at both past and future to fill gaps), JEPAhead uses a group-level causal attention mask. This forces the predictor to only look at past "tubelets" (spatio-temporal tokens) to predict future ones.
  • Residual Prediction: Instead of predicting the absolute future state, the model predicts the change (residual) relative to the last observed frame.
  • Multi-Horizon Forecasting: In a single forward pass, the model predicts 8 future token groups, covering roughly 16 frames (4 seconds of real-time driving at 4Hz).
Figure 1: Causal future prediction in the frozen V-JEPA latent space (adapted from [2]). Only the predictor Pφ is trained, with latent loss L between ˆsy and sy. (I) Context: 32 frames (16 temporal groups) of a 48-frame CARLA clip at 4 Hz. (II) Future target: 16 frames (8 groups).(III) z: 8 future queries (mask tokens + 3D positional embedding). (IV) Evaluation baselines computed from the encoder to- kens. (V) Predicted tokens decoded by GAN decoder Dβ, for image metrics. (VI) Token-space metrics.

Datasets

The study utilized the CARLA simulator, specifically Town12, which provides a complex urban environment.

  • Expert Driver: Data was collected using the PDM-Lite rule-based expert.
  • Scale: The dataset consists of over 1,700 routes, split into 70% training, 10% validation, and 20% test sets.
  • Evaluation Pool: For retrieval tasks, they used a candidate pool of 591 unique future sequences to test the model's ability to "find" the correct future.

Key Insights and Ideas

  • The "Persistence" Hurdle: In driving, the strongest baseline is often "persistence"—the assumption that nothing changes. Because most of a driving scene is static (buildings, roads), a model can look "accurate" just by repeating the last frame. JEPAhead proves its worth by significantly outperforming persistence on dynamic tokens (the parts of the scene that actually move).
  • Latent vs. Pixel Realities: The authors found that while the model excels at predicting latent features, these gains don't always translate to beautiful video. Pixel-space metrics like LPIPS showed only modest improvements. This suggests that the "bottleneck" for visual world models might be the decoder, not the predictor's understanding of the scene.
  • Semantic "Aha" Moments: By using a linear probe (a simple classifier) on the predicted tokens, they found the model captures driving-relevant information. It is particularly good at predicting the future state of vehicles, which are the most critical elements for safety.

Key Results

The "adapted" V-JEPA predictor consistently outperformed both the persistence baseline and a "compact" predictor (which had 6x fewer parameters).

  • Retrieval Accuracy: The model correctly identified the true future among 591 candidates 69.0% of the time, compared to 55.3% for persistence.
  • Horizon Gains: The model's advantage grows over time. At the shortest horizon ($h=1$), it is only 7% better than persistence; by the longest horizon ($h=8$), it is 43% better (see "ratio o/p" in the table below).
  • Vehicle Prediction: As shown in Figure 2, the model shows its highest gains in predicting patches containing vehicles, outperforming persistence by 15-22 percentage points.
Table 1: Main comparison on the test split, showing the adapted "original" model leading in cosine similarity and retrieval.
Table 2: Performance across different time horizons (h). Note how the ratio of the model's performance vs. persistence (o/p) increases as the task gets harder.

Limitations and Future Work

  • Action-Free: Currently, the model is "action-free," meaning it predicts the future based on observation but doesn't account for the ego-vehicle's intended steering or braking. This limits its use for "what-if" planning.
  • Simulation Gap: The study is restricted to CARLA. Generalization to real-world sensor data remains an open question.
  • Decoder Bottleneck: The modest gains in pixel-space metrics suggest that future work should focus on better ways to "visualize" these latent predictions or, more radically, move toward using the latents directly for downstream tasks like planning without ever decoding to pixels.
  • Object-Level Reasoning: While the model understands patches, it doesn't yet perform explicit object-level tracking or forecasting.

Summary updated

Abstract

We fine-tune a pre-trained V-JEPA1 predictor from bidirectional latent inpainting to action-free causal forecasting while keeping the encoder frozen. Using contiguous past observations, horizon-aligned targets, and group-level causal attention, the model forecasts eight future token groups, corresponding to sixteen frames, in one forward pass. On CARLA Town12, it retrieves the correct future among 591 candidates in 69.0% of cases, compared with 55.3% for persistence, with greater gains at longer horizons and on tokens exhibiting greater change. It also outperforms a compact predictor trained from scratch with six times fewer parameters. Semantic analysis indicates driving-relevant information in predicted features: a fixed linear probe trained on real encoder tokens achieves 0.16 mIoU on patches undergoing semantic-class changes, versus 0.13 for persistence. Vehicle-conditioned retrieval improves over persistence by 15 percentage points at the longest horizon. Pixel-space evaluation shows modest L1 gains but little improvement under LPIPS, motivating direct evaluation through latent and semantic measures alongside an oracle reconstruction reference. These findings support further investigation of latent forecasting for driving perception, while downstream utility remains unverified.
BibTeX
@misc{petrini2026jepahead,
  author = {Daniel G. P. Petrini and Davi Pincinato and Fabio Grassiotto and Pedro L. F. Prado and Alexandre F. S. Osorio and Raphael V. Ramos and Felipe Pamboukian},
  title = {{JEPAhead: From Inpainting to Causal Forecasting for Urban Driving}},
  year = {2026},
  eprint = {2611.08657},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08657},
}
Figure