OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observation

Henry Herzog, Favyen Bastani, Yawen Zhang, Gabriel Tseng, Joseph Redmon, Hadrien Sablon, Ryan Park, Jacob Morrison, Alexandra Buraczynski, Karen Farley, Joshua Hansen, Andrew Howe, Patrick Alan Johnson, Mark Otterlee, Ted Schmitt, Hunter Pitelka, Stephen Daspit, Rachel Ratner, Christopher Wilhelm, Sebastian Wood, Mike Jacobi, Hannah Kerner, Evan Shelhamer, Ali Farhadi, Ranjay Krishna, Patrick Beukema

Introduction

OlmoEarth introduces a new family of multimodal foundation models specifically engineered for Earth Observation (EO). By implementing a streamlined training regime known as Latent MIM Lite, researchers have achieved state-of-the-art performance across nearly 30 diverse tasks. Crucially, this approach resolves the stability issues—such as representation collapse—that frequently hinder latent-space modeling in complex visual domains.

The Challenge of Earth Observation Data

Modeling Earth Observation data presents unique difficulties compared to standard computer vision. EO data is inherently spatial (image-like), sequential (time-series of satellite passes), and highly multimodal, often requiring the integration of optical light, radar, and elevation maps.

Existing foundation models for EO face two primary hurdles:

  1. Training Instability: Advanced techniques that model images in abstract "latent space" often suffer from representation collapse, where the model fails to learn meaningful distinctions between different geographic features.
  2. The Accessibility Gap: Many cutting-edge models are computationally prohibitive for the non-profits and NGOs that require them for critical missions like conservation, food security, and disaster response.

Methodology: Latent MIM Lite

The core innovation of OlmoEarth is Latent MIM Lite (Latent Masked Image Modeling of Linear, Invariant Token Embeddings). While standard Masked Image Modeling (MIM) attempts to reconstruct missing pixels, more advanced versions try to reconstruct abstract features using a dynamic "target encoder." OlmoEarth simplifies this by using a frozen, randomly initialized linear projection to generate target features. This "Lite" approach provides the representative power of latent modeling without the instability of a moving target encoder.

The OlmoEarth pre-training pipeline, featuring multimodal satellite inputs and the Latent MIM Lite objective.

Other key technical choices include:

  • Modality-Aware Masking: To prevent the model from exploiting redundancies between different sensors (e.g., radar and optical data of the same area), OlmoEarth masks entire modalities or time-steps, forcing the model to learn deeper cross-modal relationships.
  • Bandset-Specific Loss: By restricting contrastive loss to specific subdivisions of satellite data ("bandsets"), the model avoids "easy negatives" and learns more robust, discriminative features.
  • Scalable Architecture: The models utilize a Vision Transformer (ViT) encoder-decoder architecture, ranging from "Nano" (1.4M parameters) to "Large" (300M parameters).

Global-Scale Pre-training

The model was pre-trained on a global dataset of 285,288 locations, each containing a full one-year time series. The input modalities included Sentinel-1 (radar), Sentinel-2 (multispectral), and Landsat-8. To ground the model in physical reality, it was also tasked with reconstructing derived maps such as WorldCereal (crop types), SRTM (elevation), and Canopy Height Maps.

Evaluation was conducted against 12 other foundation models using 18 research benchmarks (including GEO-Bench) and 19 real-world datasets provided by partners such as the African Wildlife Foundation and NASA JPL.

Performance and Real-World Impact

OlmoEarth establishes a new "Pareto optimum" for EO models, delivering significantly better performance for the same compute budget compared to predecessors like Clay or CROMA.

OlmoEarth establishes a new Pareto optimum for Earth Observation models, delivering superior performance across various compute scales.

Key results include:

  • Feature Extraction: The model achieved top performance on 15 out of 24 tasks when used as a frozen feature extractor (kNN/Linear Probing).
  • Fine-tuning: When fully fine-tuned, OlmoEarth led the field in 19 out of 29 tasks.
  • Case Studies: The model demonstrated immediate utility in the field, achieving a 98.1% F1 score for mangrove mapping and state-of-the-art accuracy for ecosystem classification in North Africa.

Beyond the model itself, the OlmoEarth Platform provides an end-to-end tool for non-experts to label data, fine-tune models, and run inference, directly addressing the adoption gap in the humanitarian sector.

Future Directions

While OlmoEarth represents a significant step forward, several areas for growth remain. The "Large" model did not always outperform the "Base" version, suggesting that training recipes for high-parameter EO models require further refinement. Future iterations aim to incorporate weather and climate data (such as ERA5) and higher-resolution aerial imagery to provide even more granular environmental context.

Summary updated

BibTeX
@inproceedings{herzog2026olmoearth,
  author = {Henry Herzog and Favyen Bastani and Yawen Zhang and Gabriel Tseng and Joseph Redmon and Hadrien Sablon and Ryan Park and Jacob Morrison and Alexandra Buraczynski and Karen Farley and Joshua Hansen and Andrew Howe and Patrick Alan Johnson and Mark Otterlee and Ted Schmitt and Hunter Pitelka and Stephen Daspit and Rachel Ratner and Christopher Wilhelm and Sebastian Wood and Mike Jacobi and Hannah Kerner and Evan Shelhamer and Ali Farhadi and Ranjay Krishna and Patrick Beukema},
  title = {{OlmoEarth: Stable Latent Image Modeling for Multimodal Earth Observation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=5VO4Yd97wP},
}
Figure