Forecasting Earth Surface Change in Embedding Space

Jingtao Li, Sadiq Jaffer, Anil Madhavapeddy, Pedro Sousa, Anthony Ortiz, Kyle Gao, Zhengpeng Feng, Yanfei Zhong, Kevin White, Srinivasan KeshavarXiv 2026 · arXiv

TL;DR

$E^2$Forecast shifts satellite imagery forecasting from noisy pixel-space to semantically rich embedding-space. By using a dual-branch transformer to predict both point forecasts and generative ensembles, the model achieves state-of-the-art results in forecasting accuracy and zero-shot anomaly detection across multiple Earth foundation models.

What Problem Does This Paper Address?

Predicting how the Earth's surface changes—from crop cycles to urban expansion—is vital for food security and disaster response. However, traditional satellite forecasting models typically operate in "spectral space," attempting to predict raw pixel values. This is incredibly difficult because satellite data is "noisy": clouds, shadows, and sensor artifacts fluctuate constantly, even when the underlying land remains unchanged.

Furthermore, satellite observations are rarely regular. Different sensors have different revisit times, and cloud cover creates unpredictable gaps in the data. Most existing models struggle with these irregular time series and often produce "overconfident" forecasts that fail to account for the inherent uncertainty of the future.

Methods

The authors propose $E^2$Forecast, a framework designed to forecast in the embedding space of Earth Foundation Models (GFMs). Instead of pixels, the model predicts the high-dimensional vectors that represent the "meaning" of a scene.

Key Innovations:

  1. Flexible Token Space: The model converts location (using Random Fourier Features), dates (relative order and day-of-year for seasonality), and historical embeddings into a sequence of tokens. This allows it to ingest irregular histories and query any future date.
  2. Dual-Branch Transformer:
    • Gaussian Branch: Directly predicts a mean point forecast and per-dimension standard deviations. This provides a fast, uncertainty-aware estimate.
  3. Rectified-Flow Branch: A generative component that draws realistic ensemble forecasts. This is crucial for high-dimensional embeddings where simple Gaussian sampling often fails to capture complex feature correlations.
  4. Zero-Shot Anomaly Detection: By comparing a new observation to the forecast and scaling the difference by the predicted uncertainty, the model identifies anomalies (like oil spills or dam breaks) without needing any labeled training data for those specific events.
Overview of the $E^2$Forecast architecture, showing the extraction of embeddings and the dual-branch forecasting process.

Datasets

The authors evaluated their approach on several diverse benchmarks:

  • S1_Embed: A new dataset based on Sentinel-1 (SAR) imagery across 60 global sites, using OlmoEarth embeddings.
  • S2_Embed: Based on the PASTIS dataset (Sentinel-2 optical imagery) in France, using Prithvi-EO-2.0 embeddings.
  • GSA (Global Surface Anomaly): A benchmark covering ~20,000 $km^2$ and 55 major events (e.g., fires, floods) to test anomaly detection.
  • SSL4EO-S12: Used to demonstrate the model's capability as an Implicit Neural Representation (INR) for global mapping.

Key Insights and Ideas

  • Semantic > Spectral: Forecasting embeddings is more effective because GFMs have already done the hard work of "denoising" the signal. The forecaster can focus on actual land-use changes rather than trying to predict where a cloud might appear.
  • The Generative Advantage: In high-dimensional spaces (like 1024-D embeddings), the "mean" prediction often misses fine details. The rectified-flow branch allows the model to suggest multiple "plausible futures," one of which is usually much closer to the ground truth than a simple average.
  • Forecasting as a Foundation Task: The authors show that a good forecaster is a "Swiss Army Knife." With no extra training, it can act as a global map generator (INR) or a highly accurate anomaly detector.
Figure 1. A demo of E2Forecast’s flexible forecasting with any query date, producing both point and ensemble forecasts with uncertainty. We choose this Cambridge scene because it includes both urban and rural areas. With time-series OlmoEarth embeddings, we train E2Forecast on 2024 only and use January 2025 as context, so every later forecast comes from learned temporal patterns rather than copied observations. Both point and ensemble forecasts of 10 generated embeddings capture the seasonal changes effectively, while ensemble forecasts can better describe the diversity and details.

Key Results

  • Accuracy: On Sentinel-1 and Sentinel-2 benchmarks, $E^2$Forecast halved the cosine distance error (from 0.20 to 0.10) compared to spectral-space baselines.
  • Calibration: It significantly reduced overconfidence, bringing the z-score standard deviation from a poor 2.63 down to a near-ideal 1.22.
  • Anomaly Detection: On the GSA benchmark, the model raised the $F_1$ score from 41.58 to 48.70, setting a new state-of-the-art without using any change labels or specialized detection heads.
  • Generality: The approach was validated across nine different remote sensing foundation models, proving it isn't tied to one specific architecture.

Limitations and Future Work

The authors identify two primary areas for improvement:

  1. Dimension Correlation: The current probability branch assumes embedding dimensions are independent (diagonal covariance). While the generative branch helps mitigate this, explicitly modeling cross-dimensional correlations could improve uncertainty estimates.
  2. External Drivers: Currently, the model only looks at past embeddings. It does not yet account for external "forcing" factors like weather data (rainfall, temperature) or terrain, which are major drivers of surface change. Integrating these as extra condition tokens is the planned next step.

Summary updated

Abstract

Repeated satellite imagery permits understanding of how our planet changes over time, letting us predict its future and plan mitigating actions. Most existing studies that rely on a time series of satellite observations forecast the spectral values at each pixel for a future target date. However, clouds, shadows, and sensor noise often corrupt these observations, so that spectral forecasters waste much of their capacity on low-level features or noise rather than real change. With the advent of Earth foundation models, we argue that forecasting in embedding space is a feasible and more effective alternative. To this end, we design a token space that handles irregular histories from different sensors and foundation models and supports forecasts at any query date. Tokens from this space are inputs to our proposed forecaster, E$^2$Forecast, which is a transformer with two branches: a Gaussian branch that directly predicts a mean point forecast and per-dimension standard deviations, and a rectified-flow branch that generates forecast ensembles. On Sentinel-1 and Sentinel-2 benchmarks, forecasting embeddings instead of spectral values halves the cosine distance (from 0.20 to 0.10) and greatly reduces overconfidence, lowering the z-score standard deviation from 2.63 to 1.22. By comparing new observations with its forecasts, E$^2$Forecast can also detect anomalous changes without labels or retraining, raising F$_1$ from 41.58 to 48.70 on a global benchmark. This consistent superiority is validated on nine remote sensing foundation models. Beyond change detection, we believe our approach advances the field by permitting effective reuse of diverse pretrained task heads. Our code is publicly available at https://github.com/ucam-eo/e2forecast.
BibTeX
@misc{li2026forecasting,
  author = {Jingtao Li and Sadiq Jaffer and Anil Madhavapeddy and Pedro Sousa and Anthony Ortiz and Kyle Gao and Zhengpeng Feng and Yanfei Zhong and Kevin White and Srinivasan Keshav},
  title = {{Forecasting Earth Surface Change in Embedding Space}},
  year = {2026},
  eprint = {2611.02211},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02211},
}
Figure