TL;DR
This paper benchmarks deep learning models for cloud masking on satellite imagery using the MLCommons Science benchmark. The authors present updated code, evaluate a U-Net model on NYU's HPC cluster, and report performance metrics including accuracy and training/inference times, contributing to reproducible AI for scientific applications.
What Problem Does This Paper Address?
Accurately identifying clouds in satellite imagery is crucial for many environmental applications, such as estimating sea surface temperature (SST) and land surface temperature (LST). Traditional methods like thresholding and Bayesian approaches have limitations in objectivity and flexibility. While deep learning, particularly Convolutional Neural Networks (CNNs), has shown promise, there's a need for standardized benchmarks and performance evaluations on high-performance computing (HPC) resources to drive innovation in scientific AI. This paper addresses this by evaluating the MLCommons CloudMask benchmark on an HPC cluster, providing insights into model performance and computational efficiency.
Methods
The core of this work involves evaluating a U-Net model, a popular deep learning architecture for image segmentation, on the MLCommons CloudMask benchmark. The authors leverage the reference implementation provided by MLCommons and enhance it with several key modifications:
- Improved Accuracy Calculation: Corrected and logged accuracy metrics for better tracking.
- Early Stopping: Implemented to prevent overfitting and identify the best model weights during training.
- Model Checkpointing: Saved the weights of the best-performing model.
- Reproducibility: Introduced seed-setting for pseudo-random runs to enable reporting of performance variance.
- Simplified Job Management: Developed functionality to create temporary YAML and Slurm job files for easier parallel execution and to save different models in separate directories.
The U-Net architecture itself is described as a two-part process: a contraction path to capture context and a symmetric expanding path to increase resolution and localization for pixel-wise predictions.
Datasets
The primary dataset used is the MLCommons CloudMask benchmark dataset, which consists of 180GB of Sentinel-3 satellite images. This dataset includes images captured during both day and night. For training, 90% of the data is used, with the remaining 10% reserved for testing. The images are pre-processed by cropping and dividing them into smaller 256x256 patches. Notably, the ground truth for this benchmark is derived from Bayesian masks, which differs from some other benchmarks that might use human annotations.
Key Insights and Ideas
- HPC is Essential for Scientific AI Benchmarking: The paper highlights the necessity of HPC resources like NYU Greene for handling the large datasets and computational demands of scientific AI benchmarks.
- U-Net Shows Strong Performance: The U-Net model, a standard in image segmentation, demonstrates competitive accuracy on this cloud-masking task.
- Ground Truth Matters: The authors suspect that the use of Bayesian masks as ground truth, rather than direct human annotations, might contribute to observed difficulties in training and potentially lower validation loss curves. This points to an interesting area for future research in how ground truth generation impacts model performance.
- Reproducibility and Usability are Key: The code modifications aimed at improving logging, early stopping, and job management are crucial for making scientific AI benchmarks more robust, reproducible, and accessible to the broader research community.
Key Results
The study reports that the U-Net model, when trained on the MLCommons CloudMask benchmark dataset using the NYU Greene HPC cluster, achieved a test accuracy of 0.896 in the best-performing run. The average accuracy across five experimental runs was 0.889. The training and inference times were also reported, with average training times per epoch ranging from approximately 98.75 to 142.47 minutes and average inference times per epoch around 1.41 to 1.73 minutes on V100 GPUs. The best model was identified at epoch 147 due to early stopping.
Limitations and Future Work
The authors acknowledge that the use of Bayesian masks as ground truth might present challenges for model training, potentially leading to unfavorable validation loss curves. They suggest this as a potential area for further investigation. While the paper focuses on evaluating a U-Net model, it implicitly opens the door for future work to explore other deep learning architectures or hybrid approaches for this benchmark. The updated code and benchmarking methodology also pave the way for future evaluations of more advanced models and for comparing performance across different HPC systems.
Summary updated