Agentic Building-Aware Satellite Gaussian Splatting for Auditable Urban DSM Reconstruction

Wentao Sun, Zhengsen Xu, Yiping Chen, John S. Zelek, Jonathan LiarXiv 2026 · arXiv

TL;DR

Standard neural rendering methods for satellite imagery optimize for overall visual fluff rather than the structural geometry that geospatial analysts actually care about: buildings. This paper introduces an agentic, building-aware workflow for Earth-observation Gaussian Splatting (EOGS) that uses foundation-model semantic masks to guide 3D reconstruction, overseen by an automated controller that selects, verifies, and logs the optimal reconstruction policy for every unique scene.

What Problem Does This Paper Address?

Creating accurate 3D Digital Surface Models (DSMs) from multi-view satellite imagery is notoriously difficult due to changing sun angles, seasonal variations, shadows, and transient objects. While modern neural rendering techniques like Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) produce visually stunning images, their optimization functions treat every pixel equally.

In urban planning or disaster response, a meter-level error on a building roof matters infinitely more than a visually plausible texture on a parking lot or a tree canopy. This paper addresses this asymmetry by shifting optimization capacity toward critical urban structures (buildings) without requiring tedious, manual 2D polygon annotations.

Methods

The authors propose a two-part framework that wraps around an Earth-observation Gaussian Splatting (EOGS) engine:

  1. Semantic-Weighted EOGS Optimization: The workflow generates per-view building masks using the Segment Anything Model (SAM). Instead of enforcing hard geometric constraints, these masks are injected as soft photometric weights into the loss function. Pixels labeled as "building" receive stronger supervision. The authors experiment with both static weights and a staged schedule (which gradually increases the building weight over a 5,000-iteration budget to first establish broad geometry before sharpening structural details).
  2. Agentic Reconstruction Controller: Because foundation-model masks can be imperfect or over-segment non-building regions, a "one-size-fits-all" hyperparameter fails across different scenes. The authors introduce an automated agent that acts as a quality assurance gate. It observes mask coverage, runs cheap validation checks, selects the best reconstruction policy (baseline, staged, or building-specialist), verifies the region-specific DSM metrics, and outputs an auditable JSON/CSV decision record.
Figure 1: System overview. Multi-view satellite images and SAM-derived building masks feed candidate EOGS recon- structions. The Agentic Reconstruction Controller observes mask QA and inventory metadata, acts by selecting a pol- icy, verifies region-specific DSM metrics, and reports an auditable product bundle.

Datasets

The approach was evaluated using the widely recognized DFC2019/US3D benchmark dataset, which features multi-date incidental satellite imagery accompanied by ground-truth DSMs and semantic class maps. Specifically, the authors focused on four distinct scenes: JAX_004, JAX_068, JAX_214, and JAX_260.

Key Insights and Ideas

  • Foundation Models as Task Priors: Instead of treating SAM as a standalone end-product, this paper frames foundation segmentation as a noisy, view-dependent prior to steer 3D geometry optimization.
  • The Optimization Trade-off: Aggressively optimizing for building accuracy can sometimes degrade full-scene or vegetation metrics. A staged schedule acts as an elegant compromise, improving full-scene accuracy while retaining a significant building accuracy gain.
  • Reconstruction Needs an Audit Trail: For production-grade geospatial pipelines, neural networks cannot operate as complete black boxes. By pairing the renderer with an Agentic Controller, the system documents why a certain policy was chosen based on mask quality and validation metrics, making the final 3D product auditable.
Analysis of mask quality diagnostics showing how building-mask coverage varies strongly across different scenes.

Key Results

  • Building Accuracy Gains: On the JAX_004 scene, applying a static building weight of 2.0 reduced the building-region DSM Mean Absolute Error (MAE) from 0.844 meters to 0.806 meters.
  • Balanced Global Performance: The staged schedule on JAX_004 successfully improved the full-scene MAE from 1.362 meters to 1.349 meters while still preserving a notable building-region improvement.
  • Policy Robustness: Across the remaining three JAX scenes (JAX_068, JAX_214, JAX_260), the Agentic Controller correctly identified when the SAM masks were too high-coverage or noisy and safely reverted to the validated baseline product, preventing catastrophic mask-induced reconstruction errors.
Figure 2: Building/global operating points and cross-scene regional DSM deltas. Deltas below zero indicate improvement over each baseline.

Limitations and Future Work

While the framework proves highly effective, the authors note several areas for improvement:

  • Scale of Evaluation: The method was only validated across four JAX scenes; a wider deployment on a larger Area of Interest (AOI) queue is required to fully stress-test the controller.
  • Offline Policy Selection: The current action step relies on evaluating a pre-computed candidate set of validation metrics rather than utilizing a fully generalized, learned policy network that can predict outcomes without running full trial optimizations.
  • Surface Regularization: The current pipeline outputs standard DSM/altitude geometry. Future iterations could integrate surface-aligned or 2D Gaussian Splatting variants to extract cleaner, planar mesh boundaries for roofs and walls.

Summary updated

Abstract

Urban-scale 3D reconstruction from satellite imagery supports disaster response, city monitoring, and geospatial digital twins, yet neural rendering methods typically optimize average visual fidelity rather than the structures that analysts inspect first: buildings. We present an agentic building-aware satellite Gaussian Splatting workflow that uses Segment Anything-derived building masks as semantic priors and an Agentic Reconstruction Controller to select, verify, and record DSM reconstruction policies. On the DFC2019 JAX\_004 scene, building-aware weighting reduces building-region DSM MAE from 0.844 m to 0.806 m, showing that semantic priors can shift reconstruction capacity toward analyst-critical regions. A staged schedule provides a balanced operating point, improving full-scene MAE from 1.362 m to 1.349 m while retaining a building gain. Across four JAX scenes, the Agent selects validated policies for both general DSM and building-focused DSM objectives, and produces building-inventory metadata and per-scene decision records. The system combines semantic priors, policy selection, region-specific DSM metrics, and DSM-derived GIS surface products for auditable urban 3D analysis.
BibTeX
@misc{sun2026agentic,
  author = {Wentao Sun and Zhengsen Xu and Yiping Chen and John S. Zelek and Jonathan Li},
  title = {{Agentic Building-Aware Satellite Gaussian Splatting for Auditable Urban DSM Reconstruction}},
  year = {2026},
  eprint = {2609.25578},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.25578},
}
Figure