UP-Diff: Latent Diffusion Model for Remote Sensing Urban Prediction

Zeyu Wang, Zecheng Hao, Yuhan Zhang, Yuchao Feng, Yufei Guo

TL;DR

Researchers have introduced UP-Diff, the first Latent Diffusion Model specifically designed for Remote Sensing Urban Prediction (UP). Unlike traditional change detection that looks at what has happened, UP-Diff forecasts what a city will look like by combining current satellite imagery with proposed urban planning maps, allowing planners to visualize future developments with high physical fidelity.

What Problem Does This Paper Address?

In urban planning, visualizing the impact of a new development is often a manual, expensive process. While Remote Sensing (RS) technology is advanced, most research focuses on Change Detection (CD)—identifying differences between two existing images taken at different times.

The authors identify a major gap: Change Detection is retrospective. It requires "paired" images (before and after), which are hard to collect and align. Furthermore, CD doesn't help a planner who wants to see how a specific blueprint will actually look on the ground. The authors propose the Urban Prediction (UP) task: given a "pre-change" image and a "planned change map" (a mask of where buildings or roads will go), can a model generate a realistic "post-change" image?

Methods

The authors propose UP-Diff, a framework built upon the architecture of Stable Diffusion (SD). The core innovation lies in how the model is conditioned to respect the geographic constraints of the planning map:

  1. Latent Diffusion Foundation: It uses a pre-trained autoencoder to compress images into a latent space, making the diffusion process computationally efficient.
  2. Condition Embedding:
    • Text: It uses a CLIP encoder for the prompt "A remote sensing photo" to maintain the correct visual style.
  3. Layout: It uses a trainable ConvNeXt model to encode the combination of the current satellite image and the proposed change map into "position-aware embeddings."
  4. Gated Cross-Attention: Within the UNet (the "brain" of the diffusion model), the authors added a gated mechanism. This allows the model to dynamically prioritize specific regions marked for change in the planning map, ensuring the generated buildings or roads align perfectly with the blueprint while keeping the rest of the environment (like existing forests or water) intact.
  5. Weight Transfer: By starting with pre-trained Stable Diffusion weights, the model leverages "knowledge" of textures and shapes from millions of natural images, which helps overcome the scarcity of annotated remote sensing data.

Datasets

The authors adapted two well-known Change Detection benchmarks for the new UP task:

  • LEVIR-CD: Contains high-resolution (512x512) image pairs from Texas, USA, focusing on suburban building changes.
  • SYSU-CD: A large-scale dataset from Hong Kong (256x256) containing diverse changes including buildings, roads, and even vessels on water.

In the UP task context, the "post-change" images from these datasets serve as the ground truth that the model tries to predict using only the "pre-change" image and the change mask.

Key Insights and Ideas

  • From Detection to Generation: The shift from identifying changes to synthesizing them represents a move toward "generative AI for GIS." This allows for "what-if" simulations in urban design.
  • The Power of Priors: One of the most interesting findings is that models pre-trained on natural images (like cats, dogs, and landscapes) can be effectively "steered" to generate specialized satellite imagery. This suggests that the fundamental "grammar" of visual features is shared across domains.
  • Gated Control is Crucial: Simply feeding a map into a model often leads to "hallucinations" where the model ignores the map. The gated cross-attention acts as a strict supervisor, forcing the diffusion process to attend to the specific coordinates defined in the urban plan.

Key Results

UP-Diff significantly outperformed traditional generative models (like GANs and standard VAEs) and other diffusion variants:

  • Image Quality: On the LEVIR-CD dataset, UP-Diff achieved an FID (Fréchet Inception Distance) of 117.79, a substantial improvement over the next best model (DDPM at 134.63), indicating much higher visual realism.
  • Plan Adherence: To test if the generated images actually matched the plans, the authors ran the results through a separate, pre-trained Change Detection model (DMINet). UP-Diff's generated images achieved an IoU (Intersection over Union) of 73.35% on LEVIR-CD, meaning the generated buildings were placed accurately according to the input masks.
  • Qualitative Superiority: Visual results show that while baseline models often produce blurry or "melted" building outlines, UP-Diff generates sharp, realistic structures that respect the surrounding geography (e.g., correctly placing docks near water).

Limitations and Future Work

The authors acknowledge a few areas for growth:

  • Pluralistic Prediction: Currently, the model produces one "best guess" for a future layout. Future work aims to develop "pluralistic" models that can generate multiple diverse but valid interpretations of the same plan (e.g., different architectural styles for the same building footprint).
  • Mobility and Dynamics: The current model focuses on static layouts (buildings/roads). Integrating dynamic elements like traffic patterns or seasonal vegetation changes remains a challenge.
  • Dataset Diversity: While the model performs well on the tested datasets, expanding to more diverse global climates and architectural styles is necessary for worldwide application.

Summary updated

Abstract

Remote sensing (RS) technology has become essential for monitoring urban development, including applications like population growth analysis, transportation congestion forecasting, and climate change detection (CD). However, its potential for future urban planning (UP), particularly in predicting future urban layouts remains largely unexplored. This study introduces UP-Diff, a novel method leveraging generative models for UP, to address this gap. UP-Diff leverages information from current urban layouts and planned change maps to predict future urban configurations. Key challenges addressed include the integration of urban layouts and change maps into latent diffusion model (LDM) through careful architecture improvements and the mitigation of limited training data by employing a pretrained stable diffusion (SD) model with fixed weights, trainable ConvNeXt, and trainable cross-attention layers. Our method significantly streamlines the UP process by automating layout predictions, thus reducing the time and effort required compared to traditional manual methods. Comprehensive evaluations on the learning, vision, and RS dataset (LEVIR-CD) and Sun Yat-Sen University dataset (SYSU-CD) validate that UP-Diff achieves high-fidelity predictions of future urban layouts, demonstrating its effectiveness and potential for advancing RS-based UP methodologies. Our code and model weights are available at https://github.com/zeyuwang-zju/UP-Diff.
BibTeX
@article{wang2025up,
  author = {Zeyu Wang and Zecheng Hao and Yuhan Zhang and Yuchao Feng and Yufei Guo},
  title = {{UP-Diff: Latent Diffusion Model for Remote Sensing Urban Prediction}},
  journal = {IEEE Geosci. Remote. Sens. Lett.},
  year = {2025},
  doi = {10.1109/lgrs.2024.3520133},
}
Figure