TL;DR
$E^2$Forecast shifts satellite imagery forecasting from noisy pixel-space to semantically rich embedding-space. By using a dual-branch transformer to predict both point forecasts and generative ensembles, the model achieves state-of-the-art results in forecasting accuracy and zero-shot anomaly detection across multiple Earth foundation models.
What Problem Does This Paper Address?
Predicting how the Earth's surface changes—from crop cycles to urban expansion—is vital for food security and disaster response. However, traditional satellite forecasting models typically operate in "spectral space," attempting to predict raw pixel values. This is incredibly difficult because satellite data is "noisy": clouds, shadows, and sensor artifacts fluctuate constantly, even when the underlying land remains unchanged.
Furthermore, satellite observations are rarely regular. Different sensors have different revisit times, and cloud cover creates unpredictable gaps in the data. Most existing models struggle with these irregular time series and often produce "overconfident" forecasts that fail to account for the inherent uncertainty of the future.
Methods
The authors propose $E^2$Forecast, a framework designed to forecast in the embedding space of Earth Foundation Models (GFMs). Instead of pixels, the model predicts the high-dimensional vectors that represent the "meaning" of a scene.
Key Innovations:
- Flexible Token Space: The model converts location (using Random Fourier Features), dates (relative order and day-of-year for seasonality), and historical embeddings into a sequence of tokens. This allows it to ingest irregular histories and query any future date.
- Dual-Branch Transformer:
- Gaussian Branch: Directly predicts a mean point forecast and per-dimension standard deviations. This provides a fast, uncertainty-aware estimate.
- Rectified-Flow Branch: A generative component that draws realistic ensemble forecasts. This is crucial for high-dimensional embeddings where simple Gaussian sampling often fails to capture complex feature correlations.
- Zero-Shot Anomaly Detection: By comparing a new observation to the forecast and scaling the difference by the predicted uncertainty, the model identifies anomalies (like oil spills or dam breaks) without needing any labeled training data for those specific events.
Datasets
The authors evaluated their approach on several diverse benchmarks:
- S1_Embed: A new dataset based on Sentinel-1 (SAR) imagery across 60 global sites, using OlmoEarth embeddings.
- S2_Embed: Based on the PASTIS dataset (Sentinel-2 optical imagery) in France, using Prithvi-EO-2.0 embeddings.
- GSA (Global Surface Anomaly): A benchmark covering ~20,000 $km^2$ and 55 major events (e.g., fires, floods) to test anomaly detection.
- SSL4EO-S12: Used to demonstrate the model's capability as an Implicit Neural Representation (INR) for global mapping.
Key Insights and Ideas
- Semantic > Spectral: Forecasting embeddings is more effective because GFMs have already done the hard work of "denoising" the signal. The forecaster can focus on actual land-use changes rather than trying to predict where a cloud might appear.
- The Generative Advantage: In high-dimensional spaces (like 1024-D embeddings), the "mean" prediction often misses fine details. The rectified-flow branch allows the model to suggest multiple "plausible futures," one of which is usually much closer to the ground truth than a simple average.
- Forecasting as a Foundation Task: The authors show that a good forecaster is a "Swiss Army Knife." With no extra training, it can act as a global map generator (INR) or a highly accurate anomaly detector.
Key Results
- Accuracy: On Sentinel-1 and Sentinel-2 benchmarks, $E^2$Forecast halved the cosine distance error (from 0.20 to 0.10) compared to spectral-space baselines.
- Calibration: It significantly reduced overconfidence, bringing the z-score standard deviation from a poor 2.63 down to a near-ideal 1.22.
- Anomaly Detection: On the GSA benchmark, the model raised the $F_1$ score from 41.58 to 48.70, setting a new state-of-the-art without using any change labels or specialized detection heads.
- Generality: The approach was validated across nine different remote sensing foundation models, proving it isn't tied to one specific architecture.
Limitations and Future Work
The authors identify two primary areas for improvement:
- Dimension Correlation: The current probability branch assumes embedding dimensions are independent (diagonal covariance). While the generative branch helps mitigate this, explicitly modeling cross-dimensional correlations could improve uncertainty estimates.
- External Drivers: Currently, the model only looks at past embeddings. It does not yet account for external "forcing" factors like weather data (rainfall, temperature) or terrain, which are major drivers of surface change. Integrating these as extra condition tokens is the planned next step.
Summary updated