SyntheWorld: A Large-Scale Synthetic Dataset for Land Cover Mapping and Building Change Detection

Jian Song, Hongruixuan Chen, Naoto Yokoya

TL;DR

SyntheWorld is a massive new synthetic dataset for remote sensing that leverages procedural 3D modeling and AI-generated textures (GPT-4 and Stable Diffusion) to create 40,000 high-resolution image pairs. It significantly boosts performance in land cover mapping and building change detection, particularly when real-world training data is scarce, and introduces a new metric (RDR) to predict when synthetic data will actually be beneficial.

What Problem Does This Paper Address?

In the world of computer vision, synthetic data has long been a "cheat code" for training models when real data is hard to come by. However, remote sensing (satellite and aerial imagery) has lagged behind.

The challenges are unique:

  1. Data Scarcity: Satellite imagery is often restricted by privacy, security, or high commercial costs.
  2. Annotation Bottleneck: Labeling a single 1024x1024 satellite image for land cover can take a human expert over two hours.
  3. Domain Shift: A model trained on imagery from a sunny summer in Europe often fails miserably when deployed over a rainy autumn in Southeast Asia.
  4. Scale: Unlike street-view scenes that focus on a few meters, remote sensing scenes must cover square kilometers while maintaining fine details like individual trees and building textures.

Existing synthetic datasets for remote sensing were either too small, focused on a single task (like just detecting airplanes), or lacked the stylistic diversity needed to generalize to the real world.

Methods

The authors developed a sophisticated procedural modeling pipeline in Blender to generate SyntheWorld. The workflow is broken down into four essential layers:

  • Layout: They use two methods—grid-based for regular urban/suburban areas and terrain-based (using noise textures) for irregular rural landscapes.
  • Geometry: Instead of manual modeling, they use procedural rules and "Geometry Nodes" to create infinite variations of buildings and trees.
  • Texture (The AIGC Secret Sauce): This is the most novel part. The team used GPT-4 to generate highly descriptive prompts for Stable Diffusion, which then produced over 140,000 seamless, high-quality textures for roofs, roads, and vegetation.
  • Sensor & Sunlight: They randomized camera angles (off-nadir) and sun positions to ensure the model learns to handle different lighting and perspective conditions.
The essential components for building SyntheWorld: Sensor/Sunlight, Layout, Geometry, and Texture generation.

Datasets

The paper introduces SyntheWorld, which consists of:

  • 40,000 images for land cover mapping (8 fine-grained categories).
  • 40,000 bitemporal pairs for building change detection (identifying where buildings appeared or disappeared between two time points).
  • Resolution: Sub-meter Ground Sample Distance (GSD), ranging from 0.3m to 1.0m.

To prove its worth, they evaluated SyntheWorld against several real-world benchmark datasets, including OpenEarthMap (OEM), LoveDA, WHU-CD, and LEVIR-CD+.

Key Insights and Ideas

  • AIGC for Diversity: By using generative AI for textures rather than a fixed library of images, SyntheWorld achieves a level of visual diversity that brings its feature space much closer to real-world data than previous synthetic efforts.
  • The "Low-Data" Savior: The biggest gains from synthetic data occur when you have very little labeled real-world data. Adding SyntheWorld to just 1% of a real training set provides a massive performance jump.
  • Predicting Success with RDR: The authors realized that synthetic data doesn't always help. They proposed the Relative Distance Ratio (RDR). By calculating the Fréchet Distance (a measure of similarity between data distributions) between the source, target, and synthetic domains, they can quantitatively predict if adding synthetic data will improve a model's performance.
Figure 6. Scatter diagram with correlation between mIoU changes and proposed Relative Distance Ratio (RDR).

Key Results

  • Building Segmentation: Models pre-trained on SyntheWorld outperformed those trained on all other existing synthetic remote sensing datasets (like GTA-V-SID or Synthinel-1) when tested on real-world data.
  • Land Cover Mapping: In cross-continent experiments (e.g., training on data from one continent and testing on another), SyntheWorld consistently improved mIoU (mean Intersection over Union). For example, it helped bridge the gap in difficult urban-to-rural transfers.
  • Change Detection: On the WHU-CD dataset, integrating SyntheWorld led to a nearly 10-point increase in F1 score for certain model architectures.
  • Feature Alignment: UMAP visualizations show that SyntheWorld's features cluster much more closely with real-world datasets (like OEM and LoveDA) than older synthetic datasets, which often appear as isolated outliers.
Figure 5. Qualitative results by U-Net model of continent-wise land cover mapping task.

Limitations and Future Work

Despite the progress, the authors acknowledge a persistent sim-to-real gap. While the AI-generated textures are excellent, the underlying procedural "rules" for building layouts don't always perfectly capture the chaotic reality of human urban development.

Future work will focus on refining these modeling rules to better match real-world distributions and exploring more advanced domain adaptation techniques to make the transition from SyntheWorld to the real world even more seamless.

Summary updated

Abstract

Synthetic datasets, recognized for their cost effectiveness, play a pivotal role in advancing computer vision tasks and techniques. However, when it comes to remote sensing image processing, the creation of synthetic datasets becomes challenging due to the demand for larger-scale and more diverse 3D models. This complexity is compounded by the difficulties associated with real remote sensing datasets, including limited data acquisition and high annotation costs, which amplifies the need for high-quality synthetic alternatives. To address this, we present SyntheWorld, a synthetic dataset unparalleled in quality, diversity, and scale. It includes 40,000 images with submeter-level pixels and fine-grained land cover annotations of eight categories, and it also provides 40,000 pairs of bitemporal image pairs with building change annotations for building change detection task. We conduct experiments on multiple benchmark remote sensing datasets to verify the effectiveness of SyntheWorld and to investigate the conditions under which our synthetic data yield advantages. We will release SyntheWorld to facilitate remote sensing image processing research.
BibTeX
@inproceedings{song2024syntheworld,
  author = {Jian Song and Hongruixuan Chen and Naoto Yokoya},
  title = {{SyntheWorld: A Large-Scale Synthetic Dataset for Land Cover Mapping and Building Change Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}

Loading related content...

Figure