Zero-shot building footprint extraction and regularization based on Segment Anything model with Mesh Model

Jiachen Zhong, Yongjun Zhang, Xinyi Liu, Jinming Zhang, Fei Liang, Xia Wang, Bin Zhang, Weiwei Fan, D. L. Yue2025 · DOI

Abstract

With the advancement of urbanization, building footprint data plays an important role in urban planning, 3D Real Scene, and smart cities. Traditional manual contouring methods are time-consuming and laborious, while deep learning-based building extraction methods often require a large amount of labeled data and have limited generalization ability. In this paper, a zero-shot framework based on the Segment Anything Model (SAM) is proposed for extracting and regularizing building footprints from 3D mesh data. The method mainly consists of three steps: 1) Coarse Prompt Generation, where irrelevant elements like ground and vegetation are eliminated by semi-global filtering and traditional classification, yielding a rough building mask as a bounding box prompt. 2) Fine mask generation: Using SAM's mask prompt capability, combined with logits maps and grid elevation information with adaptive thresholds to generate a fine mask prompt. This is combined with an updated bounding box to form a hybrid prompt for SAM to generate a refined building mask. 3) Footprint regularization: Kinetic Partition, Markov Random Field (MRF), and Region Growth Algorithms are used to extract regularized building contours. Structural line segments from LSD guide the Kinetic Partitioning, while MRF and region growth refine the boundaries. Our method achieved 78.31% AP50 on the Vaihingen dataset and obtained regular footprints that closely align with true building contours on real Mesh data.

1. Introduction

Accurate building footprints are the backbone of urban planning, 3D city modeling, and GIS applications. However, generating them is a notorious bottleneck. Traditional manual sketching is prohibitively expensive and slow. While deep learning models have automated much of this, they typically suffer from two major flaws: they require massive amounts of manually labeled training data, and they often fail to generalize when applied to cities with different architectural styles. Furthermore, standard segmentation models produce "blobs" (raster masks) with rounded corners, whereas real-world engineering requires regularized vector polygons with sharp, straight lines.

2. Methodology

We propose a three-stage pipeline that leverages the zero-shot capabilities of SAM without requiring task-specific retraining.

Overview of the proposed three-stage framework for building footprint extraction and regularization.

2.1 Coarse Prompt Generation

To guide SAM, the framework first processes 3D mesh data into a Digital Orthophoto Map (DOM). We use Semi-Global Filtering (SGF) to separate ground from non-ground points and a Random Forest classifier to filter out vegetation. The remaining clusters are converted into bounding box prompts. This stage effectively eliminates irrelevant elements like ground and trees before SAM is even invoked for fine segmentation.

Intermediate results of the coarse prompt generation stage: (a) initial SAM segments, (b) ground/non-ground separation, (c) classification results, and (d) final filtered building masks.

2.2 Fine Mask Generation

SAM is a 2D model and can be fooled by roofs that look like the ground. We developed a height-aware prompting strategy: we take SAM’s internal "logits map" (confidence scores) and overlay it with Digital Surface Model (DSM) height data. By using height to "nudge" SAM's confidence in ambiguous areas, we generate a much more accurate refined mask prompt. This is combined with the updated bounding box to form a hybrid prompt for the final SAM inference.

2.3 Footprint Regularization

To turn the messy raster mask into a crisp polygon, the framework uses a geometric pipeline. A Line Segment Detector (LSD) finds structural lines in the image, which guide a Kinetic Partitioning process to divide the building area into geometric tiles. Finally, a Markov Random Field (MRF) and Region Growth algorithm mathematically decide which tiles belong to the building, ensuring the final footprint has straight edges and sharp corners.

3. Results and Discussion

The framework was evaluated using the Vaihingen benchmark dataset and real-world OSGB-format mesh data.

  • Quantitative Performance: On the Vaihingen dataset, the method achieved an AP50 of 78.31% and an overall AP of 61.78%.
  • Comparative Analysis: It outperformed the popular Mask R-CNN by over 5% in Average Precision (AP) and showed a 20.81% improvement in Boundary F1 score, proving its edges are significantly more accurate.
  • Qualitative Success: On real-world mesh data, the framework successfully extracted complex building shapes where previous state-of-the-art models (like HiSup) failed to produce usable results without specific retraining.

4. Conclusion

This paper demonstrates a robust zero-shot framework for building footprint extraction. By integrating 3D height data directly into the 2D SAM prompting process and employing classical geometric algorithms for regularization, we achieve high-quality architectural footprints without the need for labeled training data. Future work will focus on improving computational speed and handling environments with extremely dense vegetation.

Summary updated

Abstract

With the advancement of urbanization, building footprint data plays an important role in urban planning, 3D Real Scene and smart cities. Traditional manual contouring methods are time-consuming and laborious, while deep learning-based building extraction methods often require a large amount of labeled data and have limited generalization ability. In this paper, a zero-shot framework based on Segment Anything Model (SAM) is proposed for extracting and regularing building footprints from 3D mesh data. The method mainly consists of three steps: 1) Coarse Prompt Generation, irrelevant element’s masks such as ground and vegetation are eliminated by semi-global filtering and traditional classification method, and rough building mask is obtained as a boundary box prompt. 2) Fine mask generation: Using SAM's mask prompt capability, combined with logits map and grid elevation information with adaptive threshold to generate the fine mask prompt. Combine it with the updated bounding box to form hybrid prompt, and input SAM to generate a refined building mask. 3) Footprint regularization: Kinetic Partition, Markov random field, and Region Growth Algorithm are used to extract regularized building contours. Structural line segments from LSD guide the Kinetic Partitioning of the building. Markov random field matches building labels, while a region growth-based boundary reassignment refines the contours. The final regularized contour integrates the partitioned building zones. Our method achieved 78.31% AP50 on the Vaihingen dataset and obtained regular footprints that closely align with the true building contours on real Mesh data.
BibTeX
@inproceedings{zhong2025zero,
  author = {Jiachen Zhong and Yongjun Zhang and Xinyi Liu and Jinming Zhang and Fei Liang and Xia Wang and Bin Zhang and Weiwei Fan and D. L. Yue},
  title = {{Zero-shot building footprint extraction and regularization based on Segment Anything model with Mesh Model}},
  booktitle = {˜The},
  year = {2025},
  doi = {10.5194/isprs-archives-xlviii-g-2025-1777-2025},
}
Figure