Vectorizing World Buildings: Planar Graph Reconstruction by Primitive Detection and Relationship Inference

Nelson Nauata, Yasutaka Furukawa

Abstract

This paper tackles a 2D architecture vectorization problem, whose task is to infer an outdoor building architecture as a 2D planar graph from a single RGB image. We provide a new benchmark with ground-truth annotations for 2,001 complex buildings across the cities of Atlanta, Paris, and Las Vegas. We also propose a novel algorithm utilizing 1) convolutional neural networks (CNNs) that detects geometric primitives and infers their relationships and 2) an integer programming (IP) that assembles the information into a 2D planar graph. While being a trivial task for human vision, the inference of a graph structure with an arbitrary topology is still an open problem for computer vision. Qualitative and quantitative evaluations demonstrate that our algorithm makes significant improvements over the current state-of-the-art, towards an intelligent system at the level of human perception.

Introduction

While human vision can easily interpret the geometric structure of a building from a top-down view, teaching computers to do the same remains a significant challenge. Most existing remote sensing algorithms focus on building footprint extraction—essentially drawing a single 1D loop around the outer boundary. However, real-world buildings have complex internal architectures, including ridges, valleys, and varying roof levels that define their 3D shape.

This research introduces a novel approach to "vectorizing" these architectures from satellite imagery, transforming raw pixels into structured 2D planar graphs (a collection of vertices and non-intersecting edges). This representation is essential for high-fidelity CAD modeling and urban planning, particularly when dealing with perspective distortion in satellite images that prevents simple "Manhattan" (right-angled) geometry assumptions.

Methodology

The proposed hybrid pipeline bridges the gap between probabilistic deep learning and hard geometric constraints through a three-stage process:

  1. Primitive Detection: Three specialized CNNs detect geometric building blocks: Corners (via FCN), Edges (via DRN), and Regions (via Mask R-CNN).
  2. Relationship Inference: The system predicts connectivity, specifically corner-to-edge relationships (which edges sprout from which corners) and region-to-region relationships (identifying shared boundaries between roof segments).
  3. Integer Programming (IP) Optimization: The IP formulation acts as the system's reasoning engine, taking noisy CNN predictions and assembling them into a single graph. It enforces "topology priors"—such as ensuring every edge has two corners and no edges cross—to guarantee a valid, clean vector map.
The proposed pipeline: from primitive detection to relationship inference and final graph reconstruction.

Dataset

To support this task, the researchers introduced a significant new benchmark consisting of 2,001 complex buildings across Atlanta, Paris, and Las Vegas, sourced from the SpaceNet corpus. Unlike standard datasets that only provide pixel masks, this dataset includes ground-truth 2D planar graphs, capturing all internal architectural feature lines. The data is split into 1,601 buildings for training and 400 for testing.

Results and Key Insights

The proposed method was compared against five state-of-the-art techniques, including PolyRNN++ and L-CNN.

  • Guaranteed Planarity: The IP-based approach achieved 100% planarity across the test set, meaning it never produced self-intersecting or "illegal" geometric graphs—a feat competing deep learning models could not match.
  • Superior Accuracy: The system achieved the highest F1-scores for corner detection (75.6) and region detection (60.8).
  • The Power of Relationships: A key finding was that adding corner-to-edge relationship constraints alone doubled the F1-score for region reconstruction, proving that local connectivity data is vital for global structural accuracy.
Reconstructed 2D planar graph showing complex internal roof architecture and successful vectorization.

Limitations and Future Work

Despite these advancements, the system faces several hurdles. It is highly dependent on the initial corner detector; if the CNN misses a corner, the IP optimizer cannot currently recover it. Additionally, the model assumes all edges are straight lines, making it unsuitable for curved architectural features. Finally, while the system is efficient for most buildings, extremely complex structures with over 50 edges can require significant optimization time. Future research will focus on improving optimization efficiency and integrating "end-to-end" learning where graph structure informs feature detection through a feedback loop.

Summary updated

Abstract

This paper tackles a 2D architecture vectorization problem, whose task is to infer an outdoor building architecture as a 2D planar graph from a single RGB image. We provide a new benchmark with ground-truth annotations for 2,001 complex buildings across the cities of Atlanta, Paris, and Las Vegas. We also propose a novel algorithm utilizing 1) convolutional neural networks (CNNs) that detects geometric primitives and infers their relationships and 2) an integer programming (IP) that assembles the information into a 2D planar graph. While being a trivial task for human vision, the inference of a graph structure with an arbitrary topology is still an open problem for computer vision. Qualitative and quantitative evaluations demonstrate that our algorithm makes significant improvements over the current state-of-the-art, towards an intelligent system at the level of human perception. We will share code and data.
BibTeX
@inproceedings{nauata2020vectorizing,
  author = {Nelson Nauata and Yasutaka Furukawa},
  title = {{Vectorizing World Buildings: Planar Graph Reconstruction by Primitive Detection and Relationship Inference}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=OyrMC08Fxif},
}
Figure