Self-Supervised Multi-Modal World Model with 4D Space-Time Embedding

Lance Legel, Qin Huang, Brandon Voelker, Daniel Neamati, Patrick Alan Johnson, Favyen Bastani, Jeff Rose, James Ryan Hennessy, Robert Guralnick, Douglas Soltis, Pamela Soltis, Shaowen Wang

TL;DR

DeepEarth is a novel self-supervised world model that introduces Earth4D, a planetary-scale 4D space-time positional encoder. By extending multi-resolution hash encoding to four dimensions (latitude, longitude, elevation, and time), the model achieves sub-meter and sub-second precision across centuries, outperforming massive foundation models on ecological forecasting tasks while using significantly less data.

What Problem Does This Paper Address?

Earth observation data is inherently multi-modal (satellite imagery, sensor logs, text reports) and deeply tied to specific points in space and time. However, existing models struggle to represent these four dimensions ($x, y, z, t$) at a global scale with high precision.

Standard positional encodings often fail to scale or lose granularity when trying to cover the entire planet over long durations. This makes it difficult for "world models"—AI systems designed to understand and simulate physical environments—to accurately fuse disparate data sources into a unified representation of our changing planet.

Methods

The authors introduce two primary components:

  1. Earth4D Encoder: Inspired by NVIDIA’s multi-resolution hash encoding (Instant-NGP), Earth4D decomposes 4D space-time into four separate 3D grids: one purely spatial ($xyz$) and three spatio-temporal ($xyt, yzt, xzt$). This decomposition allows for massively parallel computation via CUDA kernels.
  2. Learned Hash Probing: To handle "hash collisions" (where different coordinates map to the same memory location), the authors implement an end-to-end differentiable system that learns optimal memory allocation. This allows the model to "discover" meaningful patterns even when memory is constrained.
  3. DeepEarth Architecture: This is a transformer-based framework (influenced by PerceiverIO and V-JEPA) that treats multi-modal inputs as tokens. These tokens are fused with Earth4D embeddings and trained using masked reconstruction, where the model learns to fill in missing data across different modalities and time steps.

Datasets

The model was evaluated on:

  • Globe-LFMC 2.0: A global benchmark for Live Fuel Moisture Content (LFMC), a critical metric for wildfire risk assessment. It contains field measurements across diverse species and geographic regions.
  • Baseline Comparison: The authors compared DeepEarth against Galileo, a pre-trained Vision Transformer foundation model that utilizes Sentinel-2 imagery, SAR data, weather data, and topography.

Key Insights and Ideas

  • Coordinates as a Powerful Proxy: Surprisingly, Earth4D outperformed the Galileo foundation model despite Galileo having access to actual satellite imagery and weather data. Earth4D achieved this using only coordinates and species names, suggesting that high-precision 4D indexing can capture latent environmental factors (like local climate and soil) more effectively than raw pixels in some contexts.
  • Efficiency through Decomposition: By breaking 4D space into 3D sub-grids, the model maintains high resolution without the exponential memory explosion usually associated with 4D volumes.
  • The Power of Learned Probing: Integrating learned hash probing led to a 29.5% reduction in Mean Absolute Error (MAE). It allows the model to be extremely "parameter-efficient"—a 5M parameter version of the model still outperformed an 800M parameter baseline.

Key Results

  • State-of-the-Art Performance: On the LFMC benchmark, DeepEarth achieved an MAE of 11.7pp and an $R^2$ of 0.783, surpassing Galileo’s MAE of 12.6pp and $R^2$ of 0.72.
  • Extreme Compression: The model can be compressed by 99.3% (down to 5M parameters) and still maintain high accuracy, offering a 4x training speedup and 93% memory reduction compared to larger baselines.
  • High Precision: The Earth4D grid specifications allow for 24 levels of resolution, scaling from 1000km down to 0.1m spatially, and from months down to 1 second temporally.

Limitations and Future Work

  • Simulation vs. Prediction: While the paper frames DeepEarth as a "world model" capable of simulation, the primary experimental validation is focused on a regression task (LFMC prediction). Further work is needed to demonstrate its generative simulation capabilities for complex events like floods or urban growth.
  • Memory Overhead: Even with hashing, the default 24-level grid requires approximately 11GB of GPU memory during training, which may limit accessibility for researchers with smaller hardware setups.
  • Future Directions: The authors suggest that Earth4D could be integrated into even larger foundation models to provide a "universal coordinate system" for all Earth-related AI tasks.

Open Source: The code and models are available at https://github.com/legel/deepearth.

Summary updated

Abstract

We present DeepEarth, a self-supervised multi-modal world model with Earth4D, a novel planetary-scale 4D space-time positional encoder. Earth4D extends 3D multi-resolution hash encoding to include time, efficiently scaling across the planet over centuries with sub-meter, sub-second precision. Multi-modal encoders (e.g. vision-language models) are fused with Earth4D embeddings and trained via masked reconstruction. We demonstrate Earth4D's expressive power by achieving state-of-the-art performance on an ecological forecasting benchmark. Earth4D with learnable hash probing surpasses a multi-modal foundation model pre-trained on substantially more data. Access open source code and download models at: https://github.com/legel/deepearth
BibTeX
@misc{legel2026self,
  author = {Lance Legel and Qin Huang and Brandon Voelker and Daniel Neamati and Patrick Alan Johnson and Favyen Bastani and Jeff Rose and James Ryan Hennessy and Robert Guralnick and Douglas Soltis and Pamela Soltis and Shaowen Wang},
  title = {{Self-Supervised Multi-Modal World Model with 4D Space-Time Embedding}},
  year = {2026},
  eprint = {2603.07039},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.07039},
}

Loading related content...

Figure