Mask Conditional Synthetic Satellite Imagery

Van Anh Le, Varshini Reddy, Zixi Chen, Mengyuan Li, Xinran Tang, Anthony Ortiz, Simone Fobi Nsutezo, Caleb RobinsonarXiv 2023 · DOI · arXiv

Abstract

This paper explores the generation of high-fidelity synthetic Very High-Resolution (VHR) satellite imagery using land-cover masks and Generative Adversarial Networks (GANs). By addressing the "data silo" problem caused by privacy and licensing restrictions, the authors demonstrate that synthetic datasets can effectively train downstream segmentation models. Results indicate that models trained on a mixture of real and synthetic data outperform those trained on real data alone, highlighting the potential of GANs as sophisticated data augmenters in remote sensing.

Introduction

Very High-Resolution (VHR) satellite imagery is critical for applications ranging from urban planning to disaster response. However, access is often limited by legal and privacy constraints, creating a "data silo" that hinders research reproducibility. This study investigates whether synthetic imagery, conditioned on semantic layouts, can serve as a viable substitute for restricted real-world data, enabling broader collaboration in the remote sensing community.

Methodology

The researchers designed a two-stage pipeline consisting of an upstream generation task and a downstream evaluation task.

Upstream Task: Image Generation

The generation stage utilizes a conditional GAN architecture known as SPADE (Spatially Adaptive Denormalization). SPADE is specifically designed to translate semantic layouts (land-cover masks) into realistic imagery. To prevent mode collapse and ensure visual variety, a diversity loss parameter ($\lambda$) was integrated, forcing the generator to produce distinct visual outputs from the same input mask when given different noise vectors.

Downstream Task: Evaluation

To evaluate the utility of the synthetic data, a standard U-Net segmentation model was trained. The study compared performance across models trained on real imagery, synthetic imagery, and various combinations of both. The primary metric for evaluation was mean Intersection over Union (mIoU).

Overview of the upstream generation and downstream evaluation pipeline.

Datasets

The study utilized the Maryland subset of the Chesapeake Land Cover dataset, which includes 4-channel (RGB + Near-Infrared) high-resolution aerial imagery. The dataset provides 6-class land cover masks, including categories such as Water, Tree Canopy, and Impervious Road.

Results and Key Insights

The experimental results highlight several critical findings regarding the use of synthetic data in remote sensing:

  • The Importance of Diversity: Tuning the diversity loss parameter was essential for downstream utility. The researchers identified $\lambda=6$ as the optimal balance between visual fidelity and model performance.
  • Synthetic Data as a Super-Augmenter: The highest performance was achieved using a 50% real / 50% synthetic data split, which reached 0.5834 mIoU. This significantly outperformed the model trained on 100% real data (0.5235 mIoU), suggesting that the GAN provides a sophisticated form of data augmentation that improves model generalization.
  • The NIR Advantage: While 3-channel synthetic data lagged behind real data, the inclusion of the Near-Infrared (NIR) channel narrowed the gap. 4-channel synthetic imagery achieved a performance level (0.5171 mIoU) nearly identical to 3-channel real imagery.
Comparison between input class maps, generated synthetic imagery samples, and corresponding real imagery.

Conclusion and Future Work

Synthetic satellite imagery offers a promising solution to data accessibility challenges. While current methods face limitations such as tiling artifacts and the need for formal privacy verification (e.g., against membership inference attacks), the ability of synthetic data to enhance model performance through augmentation is a significant advancement. Future research will focus on scaling these methods across diverse geographic regions and evaluating performance on additional tasks like object detection.

Summary updated

Abstract

In this paper we propose a mask-conditional synthetic image generation model for creating synthetic satellite imagery datasets. Given a dataset of real high-resolution images and accompanying land cover masks, we show that it is possible to train an upstream conditional synthetic imagery generator, use that generator to create synthetic imagery with the land cover masks, then train a downstream model on the synthetic imagery and land cover masks that achieves similar test performance to a model that was trained with the real imagery. Further, we find that incorporating a mixture of real and synthetic imagery acts as a data augmentation method, producing better models than using only real imagery (0.5834 vs. 0.5235 mIoU). Finally, we find that encouraging diversity of outputs in the upstream model is a necessary component for improved downstream task performance. We have released code for reproducing our work on GitHub, see https://github.com/ms-synthetic-satellite-image/synthetic-satellite-imagery .
BibTeX
@misc{le2023mask,
  author = {Van Anh Le and Varshini Reddy and Zixi Chen and Mengyuan Li and Xinran Tang and Anthony Ortiz and Simone Fobi Nsutezo and Caleb Robinson},
  title = {{Mask Conditional Synthetic Satellite Imagery}},
  year = {2023},
  eprint = {2302.04305},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.04305},
}
Figure