TL;DR
This paper introduces Pix2Poly, a novel end-to-end trainable deep learning model that directly generates accurate polygonal building footprints from remote sensing imagery. By treating building corner detection as a sequence prediction task, Pix2Poly bypasses complex intermediate steps and achieves state-of-the-art performance with reduced model complexity.
What Problem Does This Paper Address?
Extracting precise building footprints from aerial and satellite imagery is crucial for various urban applications like mapping, 3D reconstruction, and navigation. While semantic segmentation methods can identify building pixels, they produce raster maps that often lack the geometric detail (sharp corners, straight edges) needed for these downstream tasks. Existing methods that aim for vector outputs either rely on indirect approaches with post-processing steps that can introduce errors, or direct methods that often involve complicated, non-differentiable components, hindering end-to-end training and scalability. This paper addresses the need for an efficient, scalable, and truly end-to-end trainable method that directly outputs high-quality polygonal building footprints.
Methods
Pix2Poly is an attention-based, end-to-end trainable deep neural network that generates building footprints in a ring graph format. Its core innovation lies in two main components:
- Building Vertex Sequence Detector: This module treats building corner detection as a discrete sequence prediction problem, inspired by image-to-sequence models. It uses an encoder-decoder transformer architecture. The ground truth corner coordinates are quantized into discrete tokens, forming sequences. This approach eliminates the need for non-differentiable non-maximum suppression (NMS) typically used with heatmap-based detection, ensuring end-to-end gradient flow and accurate vertex ordering.
- Optimal Matching Network: This component takes the predicted vertex sequences and their features to learn the connectivity between vertices. It predicts a permutation matrix representing the connections. The network uses both clockwise and counter-clockwise score matrices, whose transpose relationship is enforced, to ensure path consistency and generate higher-quality polygons. A differentiable Sinkhorn algorithm is used to obtain the optimal vertex assignments.
The combination of these components allows Pix2Poly to directly predict explicit, high-quality building footprints without complex rasterization losses or topological feature learning modules.
Datasets
The paper evaluates Pix2Poly on several publicly available datasets:
- Massachusetts Roads Dataset: Used for road network extraction.
- WHU Buildings Dataset: A collection of aerial images for building footprint extraction.
- INRIA Aerial Image Labelling Dataset: A widely used dataset for building segmentation, with raster ground truth that is converted to polygonal annotations for evaluation.
- SpaceNet 2: Building Detection v2 Challenge Dataset (Las Vegas subset): Provides pan-sharpened satellite imagery with polygonal building footprint annotations.
- AICrowd Mapping Challenge Dataset: Derived from the SpaceNet 2 dataset, but the authors note potential data leakage issues and move results to supplementary materials.
Key Insights and Ideas
- Sequence Prediction for Vertex Detection: Framing corner detection as a sequence prediction task is a significant departure from traditional heatmap-based approaches. This elegantly solves the problem of vertex ordering and eliminates the need for non-differentiable NMS, enabling true end-to-end training.
- End-to-End Differentiability: The entire pipeline, from image input to polygonal output, is differentiable. This simplifies training, improves scalability, and allows for more robust learning compared to methods with disconnected components.
- Reduced Model Complexity: By avoiding complex regularization modules, differentiable rasterizers, and separate topological learning, Pix2Poly achieves a lower parameter count than many state-of-the-art methods while maintaining competitive or superior performance.
- Unified Approach for Buildings and Roads: The graph-based formulation allows Pix2Poly to be extended to road network extraction, demonstrating its versatility.
Key Results
- Pix2Poly achieves state-of-the-art (SOTA) performance across various polygon quality metrics on challenging datasets like INRIA and SpaceNet Vegas.
- On the INRIA dataset, Pix2Poly achieved an IoU of 79.46% and C-IoU of 71.73%, outperforming other direct and indirect methods.
- The model exhibits significantly lower parameter complexity (31.9M) compared to methods like FFL (76.6M) and HiSup (74.3M).
- Qualitative results show Pix2Poly generating high-quality building footprints with regular corners and edges, and also producing accurate road network graphs.
Limitations and Future Work
The authors acknowledge two main limitations:
- Misalignment and Topological Errors: Occasional misalignment between predicted and ground truth polygons, and some topological errors were observed.
- Discrete Binning: The discretization of image space for tokenization might contribute to slight overlap errors.
They hypothesize that these issues could be addressed by incorporating additional regularization losses for polygon quality. Future work will focus on mitigating these limitations and further exploring the model's capabilities.
Summary updated