Zero-shot Building Attribute Extraction from Large-Scale Vision and Language Models

Fei Pan, Sangryul Jeon, Brian Wang, Frank McKenna, Stella X. Yu

TL;DR

This paper introduces a novel zero-shot workflow for extracting building attributes (like roof type, year built, number of floors) from satellite and street-view imagery. By leveraging large-scale vision-language models like CLIP and SAM, the method bypasses the need for extensive human annotations, offering improved scalability and generalization to new regions compared to existing supervised approaches.

What Problem Does This Paper Address?

Accurately identifying building attributes is crucial for tasks like urban planning, disaster preparedness, and infrastructure management. Traditional methods, such as the BRAILS framework, rely on supervised learning, which requires significant amounts of human-annotated data for each specific task (e.g., classification, segmentation). This reliance on manual annotation presents several challenges:

  • Scalability: Annotating vast datasets for diverse regions and building types is time-consuming and expensive.
  • Robustness: Models trained on specific datasets may struggle to generalize to new geographical locations with different architectural styles or image variations.
  • Annotation Imbalance: Datasets often suffer from imbalanced class distributions, making it difficult to train robust models.

The paper aims to overcome these limitations by developing a system that can extract building attributes without requiring task-specific human annotations, thereby enhancing adaptability and reducing costs.

Methods

The proposed zero-shot workflow integrates two powerful pre-trained large-scale models: CLIP (Contrastive Language-Image Pre-training) and SAM (Segment Anything Model). The workflow consists of two main components:

  1. Image-Level Captioning (for Classification): For tasks like roof type classification, the input image is processed by CLIP's image encoder. A set of relevant textual descriptions (vocabularies) for the desired attributes are then fed into CLIP's text encoder. CLIP computes the similarity between the image embedding and each text embedding, allowing it to classify the image based on the most similar textual description without any prior training on specific building attribute datasets.
  2. Segment-Level Captioning (for Segmentation): For tasks like facade parsing (identifying windows, doors, etc.), SAM is first used to segment the image into various object masks. Each masked image region is then processed by CLIP's image encoder. Similar to the image-level approach, CLIP's text encoder processes relevant textual vocabularies. The similarity scores between the masked image embeddings and text embeddings are used to assign semantic labels to each segment, effectively performing zero-shot segmentation.

This approach allows a single model to handle multiple attribute extraction tasks by simply changing the input vocabularies, a significant departure from the multi-module, supervised approach of BRAILS.

Datasets

The paper evaluates its method on datasets derived from the BRAILS framework. These include:

  • Roof Type Classification: A dataset of 6,000 labeled satellite images, with 2,000 images for each of the three common roof types: flat, gabled, and hipped.
  • Year Built Classification: A dataset of 56,660 annotated street-view images, categorized into six year-built ranges (e.g., Pre 1970, 1970-1979, etc.).
  • Facade Parsing: A dataset for segmenting building facades into semantic categories like roof, windows, and doors.
  • Number of Floors: A dataset of 60,000 street-view images from New Jersey, partitioned into training (80%), validation (15%), and testing (5%) sets.

The paper also introduces a "novel domain" for testing generalization, using street-view images from Houston, Texas, to evaluate performance on data distinct from the typical West Coast U.S. data used in BRAILS.

Key Insights and Ideas

  • Zero-Shot Powerhouse: The core insight is that large-scale, pre-trained vision-language models (CLIP, SAM) possess remarkable generalization capabilities that can be directly applied to specialized domains like building attribute extraction, bypassing the need for extensive, task-specific human annotations.
  • Unified Framework: The proposed workflow demonstrates that a single, unified model architecture can handle diverse tasks (classification, segmentation) by simply adapting the input textual vocabularies, offering a more flexible and efficient solution than traditional multi-module supervised systems.
  • Bridging the Domain Gap: The zero-shot approach shows promise in mitigating the performance degradation often seen when models are applied to new geographical regions or architectural styles, a common problem with supervised methods.
  • Semantic Matching: The method's effectiveness hinges on the ability of CLIP to semantically match visual features with textual descriptions, enabling it to understand and label building attributes based on natural language prompts.

Key Results

  • Roof Type Classification: While BRAILS achieved near-perfect accuracy (99.4%) due to training on its own validation set, the zero-shot method achieved 49.2% accuracy, showing promise and approaching BRAILS on the "flat" roof type. The paper notes the significant difference is due to BRAILS' supervised training and minimal domain gap.
  • Year Built Classification: BRAILS achieved 26.1% accuracy, while the zero-shot method achieved 20.7%. The paper highlights that this gap is acceptable given the lack of human annotations and fine-tuning for the zero-shot approach.
  • Facade Parsing: The zero-shot method achieved a mean Intersection over Union (mIoU) of 61.5%, outperforming existing open-vocabulary segmentation methods like OVSeg (57.3%) and ODISE (60.2%), demonstrating strong zero-shot segmentation capabilities.
  • Number of Floors: On BRAILS' validation set, BRAILS achieved 82.0% accuracy, while the zero-shot method achieved 75.9%. The paper notes a significant drop for three-story houses, attributing it to limited data and small visual cues.
  • Generalization: When tested on a novel domain (Houston, TX), the zero-shot method achieved 76.1% average accuracy, outperforming BRAILS (66.5%), demonstrating its superior robustness to regional variations.

Limitations and Future Work

  • Performance Gap: The paper acknowledges that for some tasks, particularly those with very specific or rare attributes (like three-story houses), the zero-shot performance may lag behind highly specialized supervised models that have been trained on extensive, domain-specific data. The performance on "three-story houses" was particularly low (0.0% for one category).
  • Vocabulary Dependence: The effectiveness of the zero-shot approach is inherently tied to the quality and relevance of the input vocabularies.
  • Future Work: The authors plan to explore more advanced utilization of large-scale models and aim to develop a specific expert model for structural and civil engineering applications. They also suggest turning the zero-shot workflow into a more advanced system.

Summary updated

Abstract

Existing building recognition methods, exemplified by BRAILS, utilize supervised learning to extract information from satellite and street-view images for classification and segmentation. However, each task module requires human-annotated data, hindering the scalability and robustness to regional variations and annotation imbalances. In response, we propose a new zero-shot workflow for building attribute extraction that utilizes large-scale vision and language models to mitigate reliance on external annotations. The proposed workflow contains two key components: image-level captioning and segment-level captioning for the building images based on the vocabularies pertinent to structural and civil engineering. These two components generate descriptive captions by computing feature representations of the image and the vocabularies, and facilitating a semantic match between the visual and textual representations. Consequently, our framework offers a promising avenue to enhance AI-driven captioning for building attribute extraction in the structural and civil engineering domains, ultimately reducing reliance on human annotations while bolstering performance and adaptability.
BibTeX
@inproceedings{pan2024zero,
  author = {Fei Pan and Sangryul Jeon and Brian Wang and Frank McKenna and Stella X. Yu},
  title = {{Zero-shot Building Attribute Extraction from Large-Scale Vision and Language Models}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=XfFPbNPRR6},
}
Figure