ZRG: A Dataset for Multimodal 3D Residential Rooftop Understanding

Isaac Corley, Jonathan Lwowski, Peyman NajafiradWACV 2024

TL;DR

This paper introduces the Zeitview Rooftop Geometry (ZRG) dataset, a large-scale, multimodal collection of residential rooftops across the US. ZRG aims to bridge the gap in existing datasets by providing high-resolution RGB imagery, digital surface models (DSMs), 3D wireframe annotations, and colored point clouds, enabling more robust research in automated rooftop analysis for applications like damage assessment and solar potential estimation.

What Problem Does This Paper Address?

Understanding the geometry and condition of residential rooftops is crucial for various applications, from predicting solar energy potential to assessing damage after natural disasters. However, existing datasets for this task are often limited in scale, resolution, or the types of data they provide. Many datasets focus on general building structures rather than specifically residential rooftops, or they lack the multimodal information (e.g., combining imagery with 3D geometry) needed for advanced analysis. This gap hinders the development of accurate and automated systems for residential rooftop understanding.

Methods

The paper introduces the ZRG dataset, which is the primary contribution. The dataset was acquired using DJI drones with high-resolution cameras, flying at low altitudes (10-15 feet) to capture detailed imagery. Multi-view reconstruction techniques were employed to generate Digital Surface Models (DSMs) and colored point clouds. For annotation, a custom tool was used by domain experts to create 3D wireframe polygons for each roof face and line geometries for edges, with 18 distinct edge categories labeled.

The authors also present baseline experiments for several common rooftop understanding tasks:

  • Roof Outline Extraction: Treated as a binary segmentation problem, evaluated using models like U-Net, U-Net++, PSPNet, and DeepLabV3+.
  • Monocular Height Estimation: Framed as a dense regression task, using similar architectures to segmentation but with a single output channel for height.
  • Planar Roof Structure Extraction: Addressed as an instance segmentation problem, utilizing a Mask R-CNN architecture.

Datasets

The paper introduces the Zeitview Rooftop Geometry (ZRG) dataset.

  • Size: Over 22,334 residential properties across the U.S.
  • Modalities: High-resolution RGB orthomosaics, Digital Surface Models (DSMs), 3D colored point clouds, and 3D roof wireframe annotations.
  • Resolution: Sub-centimeter resolution for RGB orthomosaics and DSMs.
  • Subsets: The dataset is split into ZRG-Test (1k properties), ZRG-100 (100 properties), ZRG-1k (1k properties), and ZRG-10k (10k properties) for reproducible experimentation.

The authors also compare ZRG to existing datasets like VWB, Enschede, RID, MIPD, and BuildingWF, highlighting ZRG's advantages in scale, resolution, and multimodal data.

Key Insights and Ideas

  • The Need for Large-Scale, Multimodal Data: The ZRG dataset demonstrates that a large volume of high-quality, multimodal data is crucial for achieving significant performance gains in rooftop analysis tasks, rather than just iterating on model architectures.
  • Challenges of Real-World Data: The dataset includes examples with noise from invalid pixels, overhanging vegetation, and shadows, which are acknowledged as important challenges for developing robust models.
  • Potential for 3D Reconstruction: The combination of RGB imagery, DSMs, and point clouds in ZRG opens doors for advanced 3D reconstruction of residential rooftops, enabling more detailed analysis.
  • Geographic and Property Diversity: The dataset covers various regions and property types (single-family vs. multi-family homes) across the U.S., aiming for broader applicability.

Key Results

  • Roof Outline Extraction: DeepLabV3+ with a ResNet50 backbone achieved the best performance across metrics like overall accuracy, F1 score, and mIoU.
  • Monocular Height Estimation: DeepLabV3+ with a ResNet50 backbone also outperformed other models, showing accurate pixelwise height estimation.
  • Planar Roof Structure Extraction: Performance, particularly mAP for medium-sized roof faces, significantly increased with larger training subsets (e.g., a 23.6% increase in mAP when moving from ZRG-1k to ZRG-10k). Mask R-CNN with ResNet50-FPN achieved strong results, demonstrating the ability to segment roof faces even with occlusions.
  • Dataset Size Impact: Table 2 clearly shows that larger training subsets (ZRG-10k) lead to substantially better performance in planar roof structure extraction compared to smaller subsets (ZRG-100, ZRG-1k).

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that some image acquisition stages might fail to capture the entire scene, leading to invalid pixels in the orthomosaic. They intentionally kept these properties to encourage robust model development.
  • Natural challenges like overhanging vegetation and shadows add complexity.
  • The dataset has a natural concentration of properties in the central and eastern U.S. due to client demand for roof inspections in hail-prone areas.
  • Future Work:
    • Further annotation of the dataset to include labels for roof type classification (e.g., gable, complex) and building type classification (single vs. multi-family homes).
  • Exploring recent reconstruction methods like Neural Radiance Fields (NeRF) for generating 3D models and novel views.
  • Analyzing the generalization and geographic bias of models trained on ZRG across different U.S. regions and urban/suburban areas.
  • Combining predicted outputs from different tasks (outline, height, faces) or joint learning for more comprehensive 3D reconstructions.

Summary updated

BibTeX
@inproceedings{corley2024zrg,
  author = {Isaac Corley and Jonathan Lwowski and Peyman Najafirad},
  title = {{ZRG: A Dataset for Multimodal 3D Residential Rooftop Understanding}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure