TL;DR
This paper presents a controlled, capacity-matched study comparing two dominant self-supervised pre-training objectives for video: pixel reconstruction (Autoencoders) and latent prediction (V-JEPA). The authors find a fundamental trade-off where reconstruction preserves high-fidelity physical state information, but predictive objectives produce representations that are more robust to noise and significantly more stable for long-horizon "dreaming" in latent space.
What Problem Does This Paper Address?
In the quest to build "world models" for autonomous agents, researchers often pre-train video encoders to compress visual input into useful representations. There are two main schools of thought:
- Reconstructive: Force the model to recreate the original pixels (e.g., Autoencoders).
- Predictive: Force the model to predict future features in a latent space (e.g., JEPA).
While predictive objectives are theorized to ignore "nuisance" details (like flickering lights or background noise) and focus on dynamics, it hasn't been clear how these choices actually impact downstream performance when the architecture and data are held constant. Previous comparisons often conflated the objective with differences in model scale or data quality. This paper isolates the objective to see exactly how it shapes the learned "understanding" of physics and dynamics.
Methods
The authors trained two identical ViViT (Video Vision Transformer) encoders using a time-causal attention mechanism. To ensure a fair comparison, they used a shared Finite Scalar Quantization (FSQ) bottleneck. FSQ maps continuous vectors to a finite set of discrete codes, which is known to improve stability in long-term predictions.
A key technical contribution is the asymmetric discretization scheme for the predictive (V-JEPA) model. In a standard JEPA setup, a "student" model predicts the output of a "teacher" model. The authors found that discretizing both models was problematic. Instead, they discretized the student’s representation but only applied a bounded continuous transformation to the teacher. This allowed the student to learn a discrete bottleneck while still having smooth, continuous targets to aim for.
Datasets
The study uses the DeepMind Control (DMC) Walker environment.
- Training Data: 25,000 trajectories (100 frames each) generated by a Soft Actor-Critic (SAC) agent.
- Evaluation Data: A separate held-out set of 4,165 trajectories.
- Downstream Probes: The representations were evaluated on their ability to decode the agent's physical state (positions, velocities) and actions, as well as their performance in forward dynamics modeling.
Key Insights and Ideas
- The Fidelity-Stability Trade-off: Autoencoders are "state-heavy." They excel at capturing the exact physical configuration of the agent at any given moment. However, this high fidelity comes at a cost: they are more sensitive to small errors that compound over time.
- Temporal Context is King for Prediction: JEPA representations are highly dependent on temporal context. A single "tubelet" (a small spatio-temporal patch) in a JEPA model changes significantly depending on what frames preceded it. In contrast, Autoencoder tubelets are mostly driven by local visual content.
- Inherent Noise Filtering: Predictive objectives act as a natural filter. Because random noise is unpredictable, the JEPA objective encourages the model to ignore it. This makes the resulting representations much more robust to visual perturbations like Gaussian noise compared to reconstructive models.
Key Results
The quantitative results highlight a clear divergence in representation utility:
- State Decoding: The Autoencoder (AE) achieved an $R^2$ of 0.78 for decoding physical state, nearly doubling JEPA’s 0.47.
- Long-Horizon Rollouts: Despite having slightly worse one-step prediction error, JEPA models were much more stable over time. They achieved a Compounding Ratio (CR) of 6.5, compared to the AE’s 8.0 (lower is better). This suggests JEPA is better suited for "latent dreaming" in reinforcement learning.
- Robustness: As shown in the structural probes, JEPA representations exhibited lower latent displacement and fewer changed tokens when faced with input noise, maintaining better state-probe accuracy under pressure.
Limitations and Future Work
The authors acknowledge several limitations:
- Scale: The models and datasets are relatively small compared to industry-scale foundation models.
- Environment Diversity: The study is limited to a single continuous-control environment (DMC Walker). Whether these trade-offs hold in more complex, visually cluttered, or stochastic environments remains to be seen.
- Causality: While the study identifies properties like "temporal context dependence," it doesn't strictly prove that these properties cause the improved long-horizon stability. Future work could involve intervention studies to establish these causal links.
Summary updated