Improving Satellite Imagery Masking Using Multitask and Transfer Learning

Rangel Daroya, Luísa Vieira Lucchese, Travis Simmons, Punwath Prum, Tamlin M. Pavelsky, John Gardner, Colin J. Gleason, Subhransu MajiIEEE JSTARS 2025 · DOI

TL;DR

This paper introduces a multi-task deep learning framework for simultaneously masking water, clouds, cloud shadows, terrain shadows, and snow/ice from satellite imagery. By leveraging transfer learning and a unified model, they achieve improved accuracy and significantly reduce processing time and computational resources, enabling more efficient global-scale environmental monitoring.

What Problem Does This Paper Address?

Accurate mapping of land cover and atmospheric conditions from satellite imagery is crucial for many environmental applications, such as estimating water quality variables like suspended sediment concentration (SSC). However, existing methods often rely on multiple, separate processing steps and external data sources (like digital elevation models), which can be computationally expensive and prone to errors. Specifically, standard water masking techniques often struggle with artifacts like clouds, cloud shadows, and terrain shadows, leading to inaccurate downstream analyses. The authors aim to address the need for a more accurate, efficient, and unified approach to generating these essential masks.

Methods

The core of this work is a multi-task deep learning model. Instead of training five separate models for each mask (water, cloud, cloud shadow, snow/ice, terrain shadow), a single model with a shared backbone and multiple output "heads" is trained. This allows the model to learn shared representations and improve generalization across tasks.

Key methodological aspects include:

  • Multi-task Learning: A single backbone network extracts features, which are then fed to five specialized "heads" to predict each mask simultaneously. This reduces computational overhead and can improve accuracy by leveraging inter-task relationships.
  • Transfer Learning: The model's backbone is pre-trained on large, general-purpose datasets like ImageNet, Satlas, and Prithvi, before being fine-tuned on the specific masking tasks. This helps the model learn robust feature extractors and mitigates the need for massive amounts of task-specific labeled data.
  • Architectures Explored: Both Convolutional Neural Networks (CNNs) like DeepLabv3+, MobileNetv3, SegNet, and ResNet50, and Transformer-based architectures like Vision Transformers (ViT) and Swin Transformers were investigated.

Datasets

  • Harmonized Landsat-Sentinel (HLS): This dataset provides multi-spectral satellite imagery with a high temporal resolution (2-3 days) and 30m spatial resolution. It serves as the primary input for the model.
  • Dynamic Surface Water extent (DSWx): This product, derived from HLS, provides annotated global surface water masks. Over 3 million tiles were used to train and validate the multi-task model. The DSWx labels were matched with HLS tiles, and a sampling strategy was employed to ensure temporal and spatial diversity.
  • LANA Dataset: A smaller, manually annotated dataset for cloud and cloud shadow detection was used for additional evaluation and comparison, particularly for cloud masking performance.
  • In-situ Suspended Sediment Concentrations (SSC): Data from various global water quality databases were used for the downstream application evaluation, demonstrating the impact of improved masking on SSC estimation.

Key Insights and Ideas

  • Synergy in Multi-task Learning: Predicting multiple masks simultaneously provides the model with contextual information. For instance, knowing where clouds are helps in identifying cloud shadows and can improve water and snow/ice detection by reducing false positives.
  • Transfer Learning is Crucial: Pre-training on large datasets significantly boosts performance, especially for Transformer architectures, which require more data to learn effectively compared to CNNs.
  • Architecture Trade-offs: While Transformer models like Swin-T (especially when pre-trained on Satlas) achieve top masking accuracy, they are computationally more expensive. CNN models like MobileNetv3 offer a compelling balance of accuracy and efficiency, making them suitable for applications requiring faster inference.
  • Efficiency Gains are Substantial: The multi-task approach dramatically reduces processing time and memory requirements compared to traditional, multi-step pipelines, making global-scale, frequent analysis feasible.

Key Results

  • Masking Performance: The multi-task models achieved competitive or improved F1 scores across all five masks compared to single-task equivalents. Swin-T pre-trained on Satlas showed superior performance for most masks. For water masking, Swin-T and DeepLabv3+ achieved F1 scores of 91.10% and 89.67%, respectively, outperforming baselines like MNDWI (58.43%) and DWM (82.21%) by a significant margin.
  • Efficiency: The proposed multi-task pipeline offers substantial speedups. Processing 400k samples, which takes ~86.84 days with a standard pipeline, can be reduced to ~2.78 days with a MobileNetv3 multi-task model (a 30x speedup). Peak memory consumption was also significantly reduced, with MobileNetv3 requiring less than half the RAM of the standard pipeline. Storage costs were reduced from ~96 GB for DEMs to ~0.014 GB for the MobileNetv3 model.
  • Downstream Application (SSC Estimation): Integrating the multi-task model into an SSC estimation pipeline resulted in a 2.64 mg/L reduction in RMSE and improved accuracy metrics. This highlights the practical benefit of more accurate and efficient masking.
  • Cloud Shadow Masking: While challenging, the multi-task models showed at least a 10% F1 score improvement for cloud shadow prediction compared to baselines like LANA.

Limitations and Future Work

  • DSWx Dataset Limitations: The authors acknowledge inherent limitations in the DSWx dataset, such as potential misclassifications of snow/ice in areas with high sediment concentrations or dilated cloud labels. They suggest that using manually labeled data (like LANA) for fine-tuning could improve robustness.
  • Model of a Model: The current approach relies on DSWx labels, meaning any errors in DSWx are propagated. Ideally, high-quality, manually generated labels would be used for training.
  • Future Work: The framework is broadly applicable to other masking tasks and downstream applications beyond SSC estimation. Future work could involve exploring larger datasets for pre-training, optimizing architectures for specific applications, and leveraging manually labeled data for further fine-tuning. The authors also suggest that their framework could be used to generate more accurate and efficient representations of existing products like DSWx.

Summary updated

BibTeX
@article{daroya2025improving,
  author = {Rangel Daroya and Luísa Vieira Lucchese and Travis Simmons and Punwath Prum and Tamlin M. Pavelsky and John Gardner and Colin J. Gleason and Subhransu Maji},
  title = {{Improving Satellite Imagery Masking Using Multitask and Transfer Learning}},
  journal = {IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens.},
  year = {2025},
  doi = {10.1109/jstars.2025.3551620},
}
Figure