TL;DR
This paper evaluates the Prithvi-EO-2.0-Tiny geospatial foundation model for flood mapping, finding that while pretraining significantly improves transformer performance—especially with limited data—it doesn't necessarily outperform a well-tuned, task-specific U-Net. The real "win" for compact foundation models is computational efficiency, offering nearly 2x faster inference on GPUs and 2.5x on CPUs compared to similarly sized convolutional networks.
What Problem Does This Paper Address?
Rapidly mapping surface water during floods is critical for disaster response, but creating accurate maps from satellite imagery is challenging. Traditionally, researchers have used simple spectral indices (like NDWI) or task-specific deep learning models like U-Nets.
Recently, Geospatial Foundation Models (GFMs)—large models pretrained on massive unlabeled satellite datasets—have emerged as a promising alternative. The hope is that these models can achieve higher accuracy with fewer labeled examples ("label efficiency") and generalize better to new geographic regions. However, the authors argue that the community often lacks rigorous comparisons against strong baselines. They set out to answer: Does pretraining actually beat a simple U-Net? And is the extra complexity worth the cost?
Methods
The researchers compared several architectures using the Prithvi-EO-2.0 family (developed by NASA and IBM) as their primary test subject:
- Prithvi-Tiny Variants: They tested a fine-tuned version (updating all weights), a frozen-encoder version (only updating the decoder), and a randomly initialized version (no pretraining).
- Baselines: A size-matched U-Net (7.76M parameters), a larger Prithvi-100M model, and standard spectral indices (NDWI/MNDWI).
- Evaluation Protocols:
- Protocol A: A standard train/test split where the model sees some chips from an event during training and is tested on others from the same event.
- Protocol B (LOEO): A "Leave-One-Event-Out" approach to test geographic transferability—evaluating the model on an entire flood event it has never seen before.
The workflow, including the architecture of the Prithvi segmentation network, is detailed in Figure 2:
Datasets
The study utilized the Sen1Floods11 dataset, specifically the hand-labeled Sentinel-2 subset.
- Scope: 446 image chips (512×512 pixels) covering 11 global flood events.
- Geographic Diversity: Includes events in the USA, Spain, Pakistan, India, and a specific holdout evaluation set in Bolivia.
- Input: Six multispectral bands (Blue, Green, Red, Narrow NIR, and two SWIR bands).
The geographic distribution and example chips are shown in Figure 1:
Key Insights and Ideas
- Pretraining is a "Warm Start," not a "Better Finish": Pretraining helped the Tiny transformer architecture significantly outperform its randomly initialized counterpart (a gain of ~0.046 IoU with very few labels). However, this wasn't enough to consistently beat the U-Net baseline.
- The Efficiency Dividend: The most compelling reason to use compact GFMs like Prithvi-Tiny isn't necessarily accuracy, but speed. Because the Vision Transformer (ViT) architecture is more computationally efficient for inference than the U-Net's convolutions, it processed images much faster.
- The "Boundary" Problem: Most errors occurred at the edges of water bodies. The authors suggest that the 16x16 pixel patch tokenization used in transformers might struggle with fine spatial details compared to the skip connections in a U-Net.
- Event-Level Volatility: Performance varied wildly between different flood events. A model might perform excellently in Paraguay but struggle in Pakistan, highlighting that "global" accuracy scores can hide local failures.
Key Results
- Accuracy: On the standard test set, U-Net achieved the highest IoU (0.856), followed by Prithvi-100M (0.841) and Prithvi-Tiny (0.833).
- Label Efficiency: When training data was slashed to just 25 chips, Prithvi-Tiny maintained an IoU of 0.814, showing strong resilience, though U-Net still led at 0.831.
- Inference Speed: Prithvi-Tiny was a clear winner here. It processed a 512x512 chip in 16.1 ms on a GPU (vs. 30.6 ms for U-Net) and 258 ms on a CPU (vs. 665 ms for U-Net).
- Reliability: Using predictive entropy (a measure of model uncertainty) proved highly effective at flagging erroneous pixels, with an AUROC of 0.92–0.95.
Qualitative comparisons of these results across different events can be seen in Figure 4:
Limitations and Future Work
- Domain Mismatch: The Prithvi model was pretrained on surface reflectance data (HLS), but the test dataset used top-of-atmosphere (TOA) reflectance. While normalization helps, this mismatch likely hindered the foundation model's performance.
- Spatial Resolution: The transformer's patch-based approach limits its ability to map narrow channels or complex shorelines. Future work could explore hybrid "U-Prithvi" architectures that combine pretrained encoders with convolutional skip connections.
- Temporal Data: This study focused on single-date imagery. The authors note that incorporating time-series data (pre-flood vs. during-flood) is a logical next step to better distinguish permanent water from new flooding.
Summary updated