Holistic Multi-View Building Analysis in the Wild with Projection Pooling

Zbigniew Wojna, Krzysztof Maziarz, Lukasz Jocz, Robert Paluba, Robert Kozikowski, Iasonas Kokkinos

Abstract

Analyzing building attributes from imagery is a complex task due to occlusions and viewpoint limitations. We present a novel "projection pooling" architecture that fuses aerial and street-level imagery by geometrically projecting facade features onto building footprints. Evaluated on a new large-scale dataset of nearly 50,000 images, our approach significantly improves accuracy across six fine-grained attributes by leveraging the complementary strengths of top-down and street-level perspectives.

Introduction

Accurately determining building attributes—such as construction material, number of floors, and roof geometry—is essential for diverse applications ranging from urban planning and autonomous navigation to insurance risk assessment. However, performing this analysis "in the wild" presents significant challenges. Aerial imagery provides a clear view of roof structures but often misses facade details and floor counts. Conversely, street-level imagery offers rich facade information but is frequently obstructed by trees or other structures and lacks a comprehensive view of the roof.

Existing methods typically rely on a single viewpoint or simple feature concatenation, failing to account for the complex spatial relationships between the camera's perspective and the building's physical geometry.

Methodology: The Projection Pooling Layer

To address these limitations, we introduce a deep learning architecture centered on a novel Projection Pooling layer. This approach geometrically aligns street-view features with the building's footprint in a unified top-view representation.

Multi-view building analysis framework: Aerial footprint and street-level imagery are fused to predict fine-grained attributes.

The process involves several key steps:

  1. Feature Extraction: ResNet-50 backbones extract spatial feature maps from both aerial (top-view) and multiple street-view images.
  2. The "Stripe" Approach: For each street-view image, features are averaged vertically to create a horizontal "feature stripe." This stripe captures visual information across the camera's horizontal field of view.
  3. Geometric Projection: Utilizing GPS metadata, camera orientation, and building footprint data, the model projects these stripes onto the corresponding walls of the building in the top-view grid.
  4. Fusion and Regularization: These projected features are concatenated with the aerial feature map, creating a high-dimensional tensor that encodes both visual features and their precise spatial locations. To ensure robustness, the model employs "Image Dropout" and "Cutout" techniques, forcing it to learn from redundant views and handle occlusions.

Dataset and Benchmarking

The study introduces a substantial new dataset comprising 49,426 images of 9,674 buildings globally. Sourced from Google and Bing, the dataset includes both top-view and street-view imagery. Each building is annotated across six fine-grained categories: construction type, number of floors, roof pitch, roof geometry, facade material, and occupancy class. This dataset reflects real-world complexities, including blurs, diverse architectural styles, and significant occlusions.

Key Insights and Results

The Projection Pooling model demonstrates the importance of spatial awareness, achieving an overall accuracy of 77.28%, significantly outperforming top-view only (70.10%) and street-view only (74.14%) baselines.

  • Complementary Strengths: The research quantifies how different views excel at specific tasks. Street-view imagery provides a 15% accuracy boost for floor counts, while aerial views are superior for identifying roof geometry (+3.9%).
  • Handling Redundancy: By integrating multiple street-view angles, the model effectively mitigates errors caused by occlusions. If one view is blocked, the projection pooling layer can still extract and map attributes from alternative angles.
  • Efficiency: The transition from 2D street images to a 2D top-view grid via feature stripes allows for 3D-like spatial reasoning without the computational burden of full 3D reconstruction.

The model was particularly effective at identifying Roof Geometry (91.9%) and Roof Pitch (81.3%).

Discussion and Future Work

While the model shows strong performance, it remains dependent on accurate metadata. Significant GPS or orientation errors can degrade performance, although the model includes a 20% buffer to accommodate minor imprecisions. Additionally, the vertical simplification inherent in the "stripe" approach may lose detail in buildings with vertically heterogeneous materials.

Future research directions include expanding the dataset to include object detection for smaller features like doors and windows, and applying the projection pooling concept to other multi-view domains, such as medical imaging.

Summary updated

BibTeX
@inproceedings{wojna2021holistic,
  author = {Zbigniew Wojna and Krzysztof Maziarz and Lukasz Jocz and Robert Paluba and Robert Kozikowski and Iasonas Kokkinos},
  title = {{Holistic Multi-View Building Analysis in the Wild with Projection Pooling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16393},
}
Figure