Abstract
This paper tackles a 2D architecture vectorization problem, whose task is to infer an outdoor building architecture as a 2D planar graph from a single RGB image. We provide a new benchmark with ground-truth annotations for 2,001 complex buildings across the cities of Atlanta, Paris, and Las Vegas. We also propose a novel algorithm utilizing 1) convolutional neural networks (CNNs) that detects geometric primitives and infers their relationships and 2) an integer programming (IP) that assembles the information into a 2D planar graph. While being a trivial task for human vision, the inference of a graph structure with an arbitrary topology is still an open problem for computer vision. Qualitative and quantitative evaluations demonstrate that our algorithm makes significant improvements over the current state-of-the-art, towards an intelligent system at the level of human perception.
Introduction
Current remote sensing techniques primarily focus on "building footprint extraction," which typically involves creating a 2D mask or a simple polygonal outline of a building's exterior. However, real-world architecture is far more complex, featuring internal geometries like ridges and valleys where different roof planes intersect. Inferring these structures from satellite imagery is challenging due to perspective distortions like foreshortening, which invalidate simple geometric assumptions. Furthermore, standard deep learning models often produce topologically impossible results, such as self-intersecting lines or dangling edges. This research addresses these issues by inferring a mathematically valid 2D planar graph that represents the full architectural structure of a building.
Methodology: From Pixels to Planar Graphs
The proposed hybrid pipeline bridges the gap between deep learning-based "seeing" and optimization-based "reasoning."
1. Primitive Detection
The system employs three specialized CNN architectures to detect fundamental building blocks:
- Corners: Identified using a Fully Convolutional Network (FCN).
- Edges: Detected via Dilated Residual Networks (DRN).
- Regions (Roof Planes): Segmented using Mask R-CNN.
2. Relationship Inference
Moving beyond simple object detection, the model predicts how these primitives connect. A specialized module discretizes the area around each corner into 15 angular bins to predict "corner-to-edge" relationships, while another module identifies "region-to-region" boundaries.
3. Integer Programming (IP) Assembly
The core innovation is the use of an Integer Programming (IP) solver to assemble the noisy CNN outputs. The IP solver acts as a "topological filter," maximizing confidence scores while enforcing strict geometric constraints:
- Every edge must connect two active corners.
- No two edges can intersect, ensuring planarity.
- No "dangling" edges are permitted; every corner must have at least two incident edges.
- Selected regions must be fully enclosed by edges.
Dataset and Benchmark
To evaluate the task, the authors introduced a new benchmark using high-resolution satellite imagery from the SpaceNet corpus. The dataset covers 2,001 complex buildings across Atlanta, Paris, and Las Vegas. Annotations provide ground-truth 2D planar graphs, including both external boundaries and internal architectural feature lines. The data is split into 1,601 training samples and 400 testing samples.
Key Insights and Results
A central insight of the paper is that holistic reasoning outperforms local detection. By treating "regions" (roof planes) as soft constraints, the model can infer edges even when visual signals are weak, as a valid roof plane must be bounded by edges. By offloading geometric rules to an IP solver, the authors ensure the output is a valid CAD-like vector file rather than a collection of pixels.
The proposed method significantly outperformed state-of-the-art models such as PolyRNN++, PPGNet, and L-CNN. On the Region F1-score—a metric measuring the capture of distinct roof planes—the method achieved a score of 60.8, nearly doubling the performance of PPGNet (31.6). Qualitatively, the IP-based approach consistently produced clean, planar, and topologically sound graphs that closely matched human-annotated ground truth.
Limitations and Future Work
While the system represents a significant advancement, it has limitations. It is highly dependent on initial corner detection; a missed corner cannot be recovered by the IP solver. The model also assumes piecewise linearity, which makes it difficult to represent curved architectural features accurately. Additionally, extreme lighting conditions or low contrast can cause primitive detectors to fail. Finally, while CNN inference is rapid, the IP optimization for extremely complex buildings can be computationally intensive.
Summary updated