Holistic Multi-View Building Analysis in the Wild with Projection Pooling

Zbigniew Wojna, Krzysztof Maziarz, Lukasz Jocz, Robert Paluba, Robert Kozikowski, Iasonas Kokkinos

Introduction

Remote building analysis—the process of determining a building's material, height, or usage from imagery—is a cornerstone of modern urban planning, insurance risk assessment, and disaster management. However, performing this analysis "in the wild" presents significant challenges. Aerial imagery provides a clear view of roof structures but lacks detail on facade materials or floor counts. Conversely, street-view imagery captures fine-grained facade details but is often obscured by trees, vehicles, or other urban clutter.

This paper introduces Projection Pooling, a novel deep learning architecture designed to fuse these disparate perspectives. By projecting ground-level features onto a building’s 3D footprint using geometric metadata, the authors create a unified representation that "thinks" in 3D, leading to a substantial performance boost in automated attribute classification.

Methodology: The Projection Pooling Layer

The core innovation lies in the Projection Pooling layer, which explicitly bridges the gap between 2D images and 3D physical structures. The pipeline follows a structured approach:

  1. Feature Extraction: Dual ResNet-50 backbones process top-view and street-view images independently.
  2. Stripe Generation: For street-view images, the model extracts feature maps and averages the vertical dimension to create horizontal "feature stripes." These stripes represent the building's facade as seen from a specific camera angle.
  3. Geometric Projection: Using GPS footprints and camera metadata (location, orientation, and field of view), these feature stripes are projected onto the corresponding "walls" of the building within a 2D top-view grid.
  4. Unified Representation: Projected features are max-pooled to resolve overlaps and concatenated with top-view features.
  5. Multi-task Classification: The unified map feeds six specialized classification heads.
Multi-view street-level imagery provides diverse perspectives of a single building, capturing details that aerial views might miss despite common occlusions.

To enhance robustness against real-world occlusions, the authors implemented Image Dropout, randomly omitting street-view images during training to force the model to learn from incomplete visual data.

Dataset and Annotations

To support this research, the authors released a massive new benchmarking dataset comprising 49,426 images of 9,674 unique buildings. Sampled globally to ensure architectural diversity, each building is labeled for six distinct attributes verified by professional architects and contractors.

The dataset includes expert-verified annotations for six key building attributes, ranging from construction material to occupancy type.

This dataset significantly exceeds the scale and geographic scope of previous standards like ICG Graz50 or eTRIMS, providing a more rigorous testbed for multi-view fusion models.

Results and Key Insights

The Projection Pooling model achieved an overall accuracy of 77.28%, outperforming single-view baselines and "blind" fusion techniques. Key findings include:

  • Specialized Expertise: Top-view imagery is superior for roof geometry (+3.9% accuracy), while street-view is essential for floor counts (+15% accuracy).
  • Geometric Alignment: Explicitly mapping pixels to physical walls using metadata is more effective than letting the model learn associations implicitly.
  • Redundancy Benefits: Performance improves as more street-view angles are added, though benefits plateau after 3-4 images as information overlap increases.
  • Feature Aggregation: Averaging features along the horizontal stripe was found to be more effective than summing, as it prevents the model from being biased toward walls closer to the camera.

Conclusion and Future Directions

While Projection Pooling represents a significant step forward, it remains sensitive to the accuracy of camera metadata (GPS and compass headings). Additionally, temporal mismatches between aerial and street-level captures—which may occur years apart—pose a challenge for real-world deployment. Future work may explore more granular detection, such as identifying individual windows or doors, and extending this geometric pooling technique to other multi-view domains like medical imaging.

Summary updated

BibTeX
@inproceedings{wojna2021holistic,
  author = {Zbigniew Wojna and Krzysztof Maziarz and Lukasz Jocz and Robert Paluba and Robert Kozikowski and Iasonas Kokkinos},
  title = {{Holistic Multi-View Building Analysis in the Wild with Projection Pooling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16393},
}
Figure