Self-Supervised Multi-Modal World Model with 4D Space-Time Embedding

Lance Legel, Qin Huang, Brandon Voelker, Daniel Neamati, Patrick Alan Johnson, Favyen Bastani, Jeff Rose, James Ryan Hennessy, Robert Guralnick, Douglas Soltis, Pamela Soltis, Shaowen Wang

Abstract

We present DeepEarth, a self-supervised multi-modal world model with Earth4D, a novel planetary-scale 4D space-time positional encoder. Earth4D extends 3D multi-resolution hash encoding to include time, efficiently scaling across the planet over centuries with sub-meter, sub-second precision. Multi-modal encoders (e.g. vision-language models) are fused with Earth4D embeddings and trained via masked reconstruction. We demonstrate Earth4D's expressive power by achieving state-of-the-art performance on an ecological forecasting benchmark. Earth4D with learnable hash probing surpasses a multi-modal foundation model pre-trained on substantially more data. Access open source code and download models at: https://github.com/legel/deepearth
BibTeX
@misc{legel2026self,
  author = {Lance Legel and Qin Huang and Brandon Voelker and Daniel Neamati and Patrick Alan Johnson and Favyen Bastani and Jeff Rose and James Ryan Hennessy and Robert Guralnick and Douglas Soltis and Pamela Soltis and Shaowen Wang},
  title = {{Self-Supervised Multi-Modal World Model with 4D Space-Time Embedding}},
  year = {2026},
  eprint = {2603.07039},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.07039},
}

Loading related content...

Figure