MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generation

Zhiping Yu, Chenyang Liu, Liqin Liu, Zhenwei Shi, Zhengxia ZouarXiv 2024 · OpenReview

TL;DR

MetaEarth is a novel generative foundation model that tackles the challenge of creating high-resolution, global-scale remote sensing images. It introduces a self-cascading framework and a unique noise sampling strategy to generate unbounded, multi-resolution imagery, paving the way for more realistic Earth simulations and improved data augmentation for downstream tasks.

What Problem Does This Paper Address?

Current generative models for images, while impressive, are largely limited to generating scenes at a local scale, suitable for everyday human activities. This limitation is particularly problematic for remote sensing applications, where the need for global coverage, varying resolutions, and seamless, large-scale imagery is paramount. Existing methods struggle with:

  • Scale: Generating images that cover vast geographical areas.
  • Resolution Control: Producing images with specific, controllable resolutions (e.g., meters per pixel) that reflect different imaging altitudes.
  • Unbounded Generation: Creating images of arbitrary, massive sizes without noticeable seams or discontinuities.
  • Data Diversity: The inherent diversity of Earth's landscapes (cities, forests, deserts, oceans, etc.) across different latitudes and climates poses a significant challenge for model capacity.

MetaEarth aims to overcome these limitations by enabling the generation of "virtually limitless" remote sensing images at a global scale.

Methods

MetaEarth employs a combination of novel techniques built upon the denoising diffusion model paradigm:

  • Resolution-Guided Self-Cascading Generative Framework: Instead of generating an image in a single step, MetaEarth uses a recursive, multi-stage approach. It starts with low-resolution images and progressively generates higher-resolution versions. Crucially, each stage is conditioned on the output of the previous stage and its corresponding spatial resolution. This allows for fine-grained control over resolution and content.
  • Unbounded Image Generation with Novel Noise Sampling: To address the issue of stitching multiple image blocks without visible seams, MetaEarth introduces a specialized noise sampling strategy. This strategy, combined with overlapping image blocks and a deterministic sampling process (η=0 in DDIM), ensures consistency in style and semantics across adjacent image tiles, leading to seamless, arbitrary-sized image generation.
  • Conditional Generation: The model leverages both the previously generated image and spatial resolution information as conditional inputs to guide the generation process at each stage. This is achieved through specialized encoders and upsampling layers to align feature maps.
  • High-Order Degradation for Training: To improve generalization and handle the distributional shift between training and inference in the cascaded generation process, the authors employ a high-order degradation strategy. This involves artificially degrading high-resolution images to create realistic low-resolution training pairs, mimicking the challenges of real-world data.

Datasets

  • Training Data: A large-scale dataset was constructed comprising multi-resolution optical remote sensing images (64m/pixel, 16m/pixel, and 4m/pixel) with associated geographical information (latitude, longitude, and resolution). This dataset was sourced from Google Earth and curated to remove repetitive or low-quality images. Approximately 3.1 million non-overlapping images were used for training.
  • Evaluation Data: A validation and test set were sampled from the collected data. The paper also mentions using real remote sensing images from the Sentinel-2 satellite and a "Pandora Planet" map (generated by GPT-4V) for evaluating generalization to unseen conditions.

Key Insights and Ideas

  • Cascading for Resolution Control is Key: The paper strongly advocates for a cascaded generation approach over direct high-resolution generation. This multi-stage process is essential for effectively learning and representing features at different scales and for achieving high-fidelity results.
  • Resolution as a Condition Matters: Explicitly conditioning the generation process on spatial resolution significantly improves the model's ability to perceive and represent features at various scales, leading to better image quality and more diverse outputs.
  • Noise Sampling is Crucial for Seamlessness: The proposed noise sampling strategy is vital for overcoming the visual discontinuities that arise from stitching independently generated image blocks, enabling truly unbounded image generation.
  • MetaEarth as a "Data Engine": Beyond generation, MetaEarth's ability to produce high-quality, diverse, and multi-resolution data makes it a valuable tool for data augmentation in downstream tasks, potentially improving performance on tasks with limited or sparse data.

Key Results

  • Global-Scale Generation: MetaEarth successfully generates diverse remote sensing scenes across various geographical features (glaciers, deserts, cities, farmlands, etc.) at a global scale.
  • Multi-Resolution Generation: The model can generate images with controllable resolutions, from 64m/pixel down to 4m/pixel, with clear improvements in detail and fidelity as resolution increases.
  • Unbounded Image Generation: The proposed method effectively generates large-scale, seamless images, as demonstrated by qualitative results showing no noticeable patchwork artifacts.
  • Improved Downstream Task Performance: When used for data augmentation in image classification, MetaEarth-generated data consistently improved the accuracy of various models (VGG19, ResNet34, ViT-B/32, ViT-B/16) compared to training with real data alone. For example, ResNet34 achieved an accuracy of 99.22% with augmented data versus 96.11% with real data.
  • Superiority over Text-to-Image Models: MetaEarth significantly outperforms general text-to-image models (GPT-4V, Stable Diffusion, Ernie) in generating realistic and semantically coherent remote sensing scenes, attributed to its specialized training data and architecture.

Limitations and Future Work

  • Small Object Generation: MetaEarth struggles to generate very small objects like aircraft and ships accurately, likely due to the 4m/pixel resolution limit of the training data, which leads to data imbalance issues where small objects are lost in background patterns.
  • Computational Resources: The authors acknowledge that computational constraints limit the maximum resolution of training data.
  • Future Work:
    • Achieving higher resolution image generation.
  • Developing controllable embedding for specific targets.
  • Expanding generative modalities beyond visible light to include multispectral and DEM images.
  • Further investigating controllable generation of multimodal data.
  • Exploring the potential of MetaEarth for constructing generative world models for aerospace applications, such as training intelligent agents for drones and satellites.

Summary updated

Abstract

The recent advancement of generative foundational models has ushered in a new era of image generation in the realm of natural images, revolutionizing art design, entertainment, environment simulation, and beyond. Despite producing high-quality samples, existing methods are constrained to generating images of scenes at a limited scale. In this paper, we present MetaEarth, a generative foundation model that breaks the barrier by scaling image generation to a global level, exploring the creation of worldwide, multi-resolution, unbounded, and virtually limitless remote sensing images. In MetaEarth, we propose a resolution-guided self-cascading generative framework, which enables the generating of images at any region with a wide range of geographical resolutions. To achieve unbounded and arbitrary-sized image generation, we design a novel noise sampling strategy for denoising diffusion models by analyzing the generation conditions and initial noise. To train MetaEarth, we construct a large dataset comprising multi-resolution optical remote sensing images with geographical information. Experiments have demonstrated the powerful capabilities of our method in generating global-scale images. Additionally, the MetaEarth serves as a data engine that can provide high-quality and rich training data for downstream tasks. Our model opens up new possibilities for constructing generative world models by simulating Earth visuals from an innovative overhead perspective.
BibTeX
@misc{yu2024metaearth,
  author = {Zhiping Yu and Chenyang Liu and Liqin Liu and Zhenwei Shi and Zhengxia Zou},
  title = {{MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generation}},
  year = {2024},
  eprint = {2405.13570},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.13570},
}
Figure