TerraMind: Large-Scale Generative Multimodality for Earth Observation

Johannes Jakubik, Felix Yang, Benedikt Blumenstiel, Erik Scheurer, Rocco Sedona, Stefano Maurogiovanni, Jente Bosmans, Nikolaos Dionelis, Valerio Marsocci, Niklas Kopp, Rahul Ramachandran, Paolo Fraccaro, Thomas Brunschwiler, Gabriele Cavallaro, Juan Bernabe-Moreno, Nicolas Longépé

TL;DR

TerraMind is a novel, large-scale generative multimodal model for Earth Observation (EO) that can generate any type of EO data from any other type. It achieves state-of-the-art performance by leveraging a unique dual-scale approach (pixel and token level) and a "Thinking-in-Modalities" technique for enhanced generation.

What Problem Does This Paper Address?

Earth Observation (EO) data is incredibly rich, coming from various sensors (optical, radar, elevation, etc.) and often accompanied by textual descriptions. However, integrating these diverse data sources (modalities) effectively for tasks like generation and analysis remains a significant challenge. Existing models often focus on specific modalities or tasks, and struggle to generalize across different data types. Furthermore, EO data can be sparse due to factors like weather or sensor revisit times, making it difficult to obtain complete information. This paper aims to bridge this gap by creating a single, powerful model that can understand and generate across a wide range of EO modalities, effectively filling in missing information and enabling new applications.

Methods

TerraMind employs a two-stage pre-training approach.

  1. Unimodal Tokenization: For each modality (e.g., Sentinel-2 images, radar data, elevation models, land cover maps, text captions), a specialized tokenizer is trained. Image-like modalities are processed using autoencoder-based architectures with a quantization step (Finite Scalar Quantization - FSQ) to convert image patches into discrete tokens. Textual data (captions and geolocations) is handled by a standard text tokenizer. This tokenization allows for a unified representation across different data types.
  2. Dual-Scale Pre-training: The core of TerraMind is its encoder-decoder architecture, which is pre-trained using a masked modeling objective. Crucially, it uses a "dual-scale early fusion" approach. This means it processes both:
  3. Token-level inputs: High-level contextual information derived from the tokenizers.
  4. Pixel-level inputs: Fine-grained spatial details directly from the raw sensor data. This dual-scale approach allows the model to learn correlations both within and across modalities at different levels of abstraction.

A key innovation is the "Thinking-in-Modalities" (TiM) technique. This allows the model to recursively generate synthetic modalities during fine-tuning or inference, effectively augmenting the input data and improving performance, especially in scenarios with missing information.

Datasets

The primary dataset used for training TerraMind is TerraMesh, a custom, large-scale multimodal dataset comprising approximately 9 million samples. TerraMesh is built upon existing datasets like SSL4EO-S12 and MajorTOM-Core, and expands them by incorporating:

  • Sentinel-1 (radar) and Sentinel-2 (optical) satellite imagery.
  • Land Use/Land Cover (LULC) maps.
  • Normalized Difference Vegetation Index (NDVI) maps.
  • Digital Elevation Models (DEM).
  • Geographic coordinates.
  • Natural language captions.

For evaluation, the paper uses the PANGAEA benchmark for fine-tuning experiments and standard datasets like EuroSAT and METER-ML for few-shot learning.

Key Insights and Ideas

  • Dual-Scale Fusion is Crucial: Combining both token-level (high-level context) and pixel-level (fine-grained detail) representations is significantly more effective than using either alone, leading to better structured embedding spaces and improved performance.
  • "Thinking-in-Modalities" (TiM) is Powerful: The ability to generate synthetic data during fine-tuning or inference acts like a "chain-of-thought" for EO models, allowing them to reason over missing information and boost performance. This is particularly valuable for tasks where data is incomplete.
  • Any-to-Any Generative Capability: TerraMind is not limited to specific generation tasks; it can generate any modality from any other modality, opening up a wide range of novel applications.
  • Tokenization Enables Modality Agnosticism: By converting diverse data into a common token format, TerraMind can be extended to new modalities simply by developing a new tokenizer.

Key Results

  • State-of-the-Art Performance: TerraMind achieves beyond state-of-the-art performance on community-standard benchmarks, outperforming other geospatial foundation models in both unimodal and multimodal settings. For instance, on the PANGAEA benchmark, TerraMindv1-B outperforms other GeoFMs by at least 3pp avg. mIoU.
  • Zero-Shot Capabilities: TerraMind demonstrates impressive zero-shot performance on tasks like water body mapping (achieving 45.4% IoU with zero-shot S-2 input, compared to SOTA fine-tuning of 82.2%) and geo-localization.
  • Few-Shot Learning: In 1-shot 5-way classification experiments on EuroSAT, TerraMind outperforms other benchmarks by at least 10pp in accuracy, indicating a well-structured latent space.
  • TiM Benefits: Using TiM tuning for water body mapping improved performance by up to 2pp mIoU compared to standard fine-tuning.
  • Generative Quality: Quantitative evaluations show reasonable quality for radar-to-optical generation, while optical-to-radar generation yields visually plausible but structurally less aligned outputs. DEM generation is structurally strong but can be noisy.

Limitations and Future Work

  • Generative Asymmetry: The paper notes an asymmetry in generative performance, with radar-to-optical generation being more robust than optical-to-radar in terms of structural fidelity.
  • DEM Altitude Inference: The model struggles with inferring altitude for DEM generation, leading to noisy outputs.
  • Future Directions: The authors suggest extending TerraMind to multi-temporal, multi-resolution, and hyperspectral data to further leverage the rich EO data ecosystem. The "Thinking-in-Modalities" approach also holds promise for further exploration.

Summary updated

Abstract

We present TerraMind, the first any-to-any generative, multimodal foundation model for Earth observation (EO). Unlike other multimodal models, TerraMind is pretrained on dual-scale representations combining both token-level and pixel-level data across modalities. On a token level, TerraMind encodes high-level contextual information to learn cross-modal relationships, while on a pixel level, TerraMind leverages fine-grained representations to capture critical spatial nuances. We pretrained TerraMind on nine geospatial modalities of a global, large-scale dataset. In this paper, we demonstrate that (i) TerraMind's dual-scale early fusion approach unlocks a range of zero-shot and few-shot applications for Earth observation, (ii) TerraMind introduces "Thinking-in-Modalities" (TiM) -- the capability of generating additional artificial data during finetuning and inference to improve the model output -- and (iii) TerraMind achieves beyond state-of-the-art performance in community-standard benchmarks for EO like PANGAEA. The pretraining dataset, the model weights, and our code are open-sourced under a permissive license.
BibTeX
@inproceedings{jakubik2025terramind,
  author = {Johannes Jakubik and Felix Yang and Benedikt Blumenstiel and Erik Scheurer and Rocco Sedona and Stefano Maurogiovanni and Jente Bosmans and Nikolaos Dionelis and Valerio Marsocci and Niklas Kopp and Rahul Ramachandran and Paolo Fraccaro and Thomas Brunschwiler and Gabriele Cavallaro and Juan Bernabe-Moreno and Nicolas Longépé},
  title = {{TerraMind: Large-Scale Generative Multimodality for Earth Observation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure