TL;DR
This paper introduces Re:PolyWorld, a novel graph neural network architecture that improves polygonal scene parsing by jointly considering vertex and edge features. It generalizes the polygonal scene representation to handle complex scenarios with shared vertices, achieving state-of-the-art results in building extraction, floorplan reconstruction, and wireframe parsing.
What Problem Does This Paper Address?
Many computer vision applications, such as architectural reconstruction and autonomous driving, require precise vector representations of objects (polygons) rather than pixel-based masks. Traditional methods often struggle with complex scenes where objects share vertices, leading to inaccurate or oversimplified polygon outputs. Existing approaches, like the original PolyWorld, are limited by their inability to handle these shared vertex scenarios, restricting their applicability. This paper aims to overcome these limitations by proposing a more generalized and robust polygonal scene representation and a novel neural network architecture to achieve it.
Methods
The core of Re:PolyWorld is a graph neural network (GNN) that processes image data to extract and connect object vertices into polygons. The architecture consists of three main components:
- Graph Encoder Network (GEN): This module uses a CNN backbone to extract local vertex features and edge features from an input image. It predicts vertex locations and generates initial visual descriptors for both vertices and edges.
- Edge-Aware Graph Neural Network (EA-GNN): This is the novel component. It enhances the initial vertex and edge representations by propagating information globally through a graph structure. Crucially, it incorporates an "edge-aware self-attention mechanism" that allows the network to reason about the visual appearance of edges, which helps disambiguate vertex connections, especially in cases of self-similar vertices. This module updates vertex and edge representations by aggregating messages across the graph.
- Optimal Connection Network (OCN): This module takes the enriched vertex and edge descriptors and predicts the connections between vertices to form polygons. It uses a differentiable optimal transport approach (specifically, the Sinkhorn algorithm) to solve a linear sum assignment problem, finding the optimal set of connections. A key innovation here is the generalized polygonal scene representation, which uses "multi-instance vertices" (allowing a vertex to have multiple connections) and an adjacency matrix, which can be expanded into a permutation matrix. This allows the model to handle polygons with shared vertices, a limitation of previous methods. The paper also introduces a "double-path consistency" mechanism to ensure robustness in clockwise and counter-clockwise vertex ordering.
Datasets
The paper evaluates Re:PolyWorld on several datasets:
- CrowdAI Mapping Challenge dataset: Used for building extraction and polygonization.
- S3D dataset: Used for floorplan reconstruction.
- HEAT dataset: Used for outdoor building reconstruction.
- Wireframe Parsing dataset: Used for wireframe parsing.
The authors note that their approach is trained in a fully supervised manner, requiring high-quality vector annotations.
Key Insights and Ideas
- Edge features are crucial for disambiguation: The paper highlights that simply relying on vertex information is insufficient. The visual appearance of edges provides critical cues for correctly connecting vertices, especially in complex scenes.
- Generalized polygonal representation: The introduction of multi-instance vertices and the adjacency matrix representation overcomes the limitations of permutation-based representations, enabling the model to handle polygons with shared vertices. This generalization is key to applying the method to diverse tasks.
- Edge-aware attention for global reasoning: The EA-GNN with its attention mechanism allows the network to leverage global scene context and edge information, leading to more accurate and refined polygon predictions.
- End-to-end training with optimal transport: The use of a differentiable optimal transport loss for connection prediction allows for end-to-end training, simplifying the pipeline and improving performance.
Key Results
- Building Extraction: Re:PolyWorld outperforms the original PolyWorld and other state-of-the-art methods on the CrowdAI dataset, achieving higher instance precision and recall, as well as improved footprint quality metrics (Complexity-aware IoU and MTA).
- Floorplan & Outdoor Reconstruction: The model demonstrates strong generalization capabilities on the S3D and HEAT datasets, producing precise and visually accurate polygons, especially when the positional refinement offset is enabled.
- Wireframe Parsing: Re:PolyWorld achieves state-of-the-art performance in wireframe parsing, generating a significantly lower number of proposals compared to other methods while maintaining high accuracy. The paper notes that its approach, by solving an optimal transport problem, naturally leads to fewer proposals.
Limitations and Future Work
- Supervised Learning Dependency: The model requires high-quality vector annotations for training. Noisy or misaligned annotations can lead to incomplete polygons or the omission of entire objects.
- Handling of Undesired Behavior: The paper notes that in cases of poor annotations, the network might produce polygons with missing corners or discard the entire object footprint. Visual examples of this are provided in the supplementary material.
Future work could explore methods to mitigate the impact of noisy annotations or investigate semi-supervised or unsupervised learning approaches for polygonal scene parsing. The generalized representation also opens doors for applying the model to a wider range of geometric reconstruction tasks.
Summary updated