Prompt-Driven Building Footprint Extraction in Aerial Images with Offset-Building Model

Kai Li, Yupeng Deng, Yunlong Kong, Diyou Liu, Jingbo Chen, Yu Meng, Junxian Ma, Wang, ChenhaoarXiv 2023 · DOI · arXiv

TL;DR

This paper introduces the Offset-Building Model (OBM), a novel prompt-driven approach for extracting building footprints from aerial imagery. By leveraging the power of models like SAM and introducing a new offset prediction mechanism, OBM significantly improves accuracy and generalization, making large-scale building footprint extraction more efficient and cost-effective.

What Problem Does This Paper Address?

Extracting accurate building footprints from aerial images is crucial for various applications like 3D modeling and change detection. Traditional methods often struggle with generalization to large datasets and require significant manual effort for correction. Specifically, extracting building footprints from off-nadir (angled) aerial images is challenging because the roof and footprint are not directly aligned, leading to small offsets that are hard to detect. Existing deep learning methods, while powerful, can suffer from poor generalization and unstable post-processing steps (like Non-Maximum Suppression or NMS), making them difficult for real-world data production. The paper aims to bridge this gap by developing a promptable framework that allows for more intuitive and accurate building footprint extraction, even with challenging offsets.

Methods

The core of this work is the Offset-Building Model (OBM), which builds upon the capabilities of the Segment Anything Model (SAM). OBM introduces a Reference Offset Augment Module (ROAM) specifically designed to predict roof-to-footprint offsets. ROAM uses "offset tokens" and a multi-head structure (Base Head and Adaptive Heads) to refine offset predictions, especially for buildings with varying offset lengths.

To address the limitations of traditional NMS, the authors propose Distance-NMS (D-NMS) and soft Distance-NMS (soft D-NMS). These algorithms refine offset predictions by considering the length of the offsets, aiming to improve directional accuracy without discarding potential building instances.

The paper also introduces a prompt-based evaluation method to better assess the performance of prompt-driven models.

Datasets

  • BONAI dataset: Used for training and evaluation. It contains 3,000 images for training/validation and 300 for testing, with a resolution of 1024x1024.
  • Huizhou test set: A newly proposed dataset with over 7,000 manually annotated instance samples, used to validate the generalization capabilities of the models. This dataset features images with a spatial resolution of 0.5m.
  • OmniCity-view3 dataset: Used for further evaluation, containing 17,092 training and 4,929 testing images (512x512 resolution) with offset annotations.

Key Insights and Ideas

  • Prompting is key for generalization: By shifting from end-to-end models to prompt-driven approaches, the models become more adaptable and generalize better to unseen data and varying conditions.
  • Offset prediction is crucial for off-nadir imagery: Accurately predicting the offset between a roof and its footprint is essential for extracting building outlines from angled aerial views.
  • Leveraging attention for context: OBM's transformer-based architecture allows it to consider features from surrounding buildings, providing valuable context that helps predict offsets, especially for shorter ones.
  • DNMS for better offset refinement: The proposed DNMS algorithms effectively improve the directional accuracy of predicted offsets by leveraging patterns observed in longer offsets.
  • New evaluation metrics are needed: Traditional metrics might not fully capture the performance of prompt-level models, highlighting the need for specialized evaluation methods.

Key Results

  • OBM reduces offset errors by 16.6% and improves roof Intersection over Union (IoU) by 10.8% compared to other models on the BONAI dataset.
  • DNMS algorithms further reduce offset vector loss by 6.5%.
  • On the new Huizhou test set, OBM achieved 20.29% lower vector loss and 76.36% lower angle loss compared to previous methods.
  • OBM demonstrates superior performance in predicting building footprints, achieving higher F1-scores and better offset metrics across various datasets.
  • The model can generate relative height maps based on roof segmentation and offset predictions.

Limitations and Future Work

  • SAM's inherent limitations: OBM inherits SAM's limitation regarding mask boundaries, which may not always be perfectly aligned with buildings. Future work will focus on improving these boundaries.
  • Handling significant offsets: The models still struggle with buildings that have very large roof-to-footprint offsets, as these are scarce in training data.
  • Exploring alternative structures: The paper suggests that combining different types of segmentation (roof, footprint, building body) could lead to further improvements in offset and footprint prediction.
  • Iterative soft DNMS: The paper hints at the potential for iteratively applying soft DNMS for even better offset quality.

Summary updated

Abstract

More accurate extraction of invisible building footprints from very-high-resolution (VHR) aerial images relies on roof segmentation and roof-to-footprint offset extraction. Existing methods based on instance segmentation suffer from poor generalization when extended to large-scale data production and fail to achieve low-cost human interaction. This prompt paradigm inspires us to design a promptable framework for roof and offset extraction, and transforms end-to-end algorithms into promptable methods. Within this framework, we propose a novel Offset-Building Model (OBM). Based on prompt prediction, we first discover a common pattern of predicting offsets and tailored Distance-NMS (DNMS) algorithms for offset optimization. To rigorously evaluate the algorithm's capabilities, we introduce a prompt-based evaluation method, where our model reduces offset errors by 16.6\% and improves roof Intersection over Union (IoU) by 10.8\% compared to other models. Leveraging the common patterns in predicting offsets, DNMS algorithms enable models to further reduce offset vector loss by 6.5\%. To further validate the generalization of models, we tested them using a newly proposed test set, Huizhou test set, with over 7,000 manually annotated instance samples. Our algorithms and dataset will be available at https://github.com/likaiucas/OBM.
BibTeX
@misc{li2023prompt,
  author = {Kai Li and Yupeng Deng and Yunlong Kong and Diyou Liu and Jingbo Chen and Yu Meng and Junxian Ma and Wang, Chenhao},
  title = {{Prompt-Driven Building Footprint Extraction in Aerial Images with Offset-Building Model}},
  year = {2023},
  eprint = {2310.16717},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.16717},
}

Loading related content...

Figure