TOWARDS LARGE-SCALE BUILDING ATTRIBUTE MAPPING USING CROWDSOURCED IMAGES: SCENE TEXT RECOGNITION ON FLICKR AND PROBLEMS TO BE SOLVED

Yao Sun, Anna Kruspe, Liqiu Meng, Yajie Tian, Eike Jens Hoffmann, Stefan Auer, Xinhua Zhu2023 · DOI

Abstract

Crowdsourced platforms provide huge amounts of street-view images that contain valuable building information. This work addresses the challenges in applying Scene Text Recognition (STR) in crowdsourced street-view images for building attribute mapping. We use Flickr images, particularly examining texts on building facades. A Berlin Flickr dataset is created, and pre-trained STR models are used for text detection and recognition. Manual checking on a subset of STR-recognized images demonstrates high accuracy. We examined the correlation between STR results and building functions, and analysed instances where texts were recognized on residential buildings but not on commercial ones. Further investigation revealed significant challenges associated with this task, including small text regions in street-view images, the absence of ground truth labels, and mismatches in buildings in Flickr images and building footprints in OpenStreetMap (OSM). To develop city-wide mapping beyond urban hotspot locations, we suggest differentiating the scenarios where STR proves effective while developing appropriate algorithms or bringing in additional data for handling other cases. Furthermore, interdisciplinary collaboration should be undertaken to understand the motivation behind building photography and labeling. The STR-on-Flickr results are publicly available.

Introduction

Urban mapping traditionally relies on satellite and aerial imagery. However, these "nadir-looking" (top-down) views primarily capture rooftops, leaving building functions, ages, and ownership—details visible only on facades—shrouded in ambiguity. While Google Street View (GSV) offers a solution, it is a commercial product with significant usage restrictions. Crowdsourced platforms like Flickr provide a massive, free alternative, but the data is inherently "noisy." Images are not always of buildings, GPS tags can be inaccurate, and text on facades is often small, distorted, or occluded. This research investigates whether state-of-the-art Scene Text Recognition (STR) models can reliably extract building information from this messy, crowdsourced data to enrich OpenStreetMap (OSM).

Methodology

The researchers developed a multi-stage pipeline to transform raw Flickr uploads into structured building data:

  1. Image Filtering: To identify relevant street-view-style images, a VGG16 network was used to calculate cosine similarity between Flickr images and a "seed" dataset of GSV images. This was further refined using object detection to ensure a building facade was present.
  2. Geospatial Mapping: Using EXIF metadata (GPS and compass direction), the team projected a "line of sight" to match images to specific building footprints in OpenStreetMap (OSM).
  3. Scene Text Recognition (STR): Two specialized models were employed: TextSnake for text detection (capable of handling curved or irregularly shaped text) and SAR (Show, Attend and Read) for character recognition.
  4. Post-Processing: Results were filtered to remove "noise," such as stopwords, repetitive architectural patterns misidentified as text (e.g., balcony railings), and low-confidence detections.
Complex scene text recognition on a large commercial building in Berlin showing multiple detections after filtering.

Key Insights and Results

The study utilized a custom Berlin Flickr Dataset, starting with over 900,000 images and filtering down to 3,431 high-quality street-view building images. Valid text was successfully recognized and filtered on 1,558 images (45.4%).

The "Residential" Paradox

Surprisingly, many buildings labeled "residential" in OSM yielded high STR results. This revealed that many residential buildings have ground-floor commercial uses (cafes, pharmacies) not captured in the primary OSM tag. Conversely, many "commercial" buildings lacked recognizable text because they rely on brand logos or visual cues (like mannequins) that STR models are not designed to "read."

Accuracy and Domain Gaps

In a manual check of a subset of images where only numbers were detected, the STR system achieved 90.62% accuracy. These numbers primarily represented house numbers and construction years. However, the researchers noted a significant "domain gap": most STR models are trained on text-centric images where the text is the main subject. In crowdsourced street views, text regions are often tiny relative to the whole image, leading to frequent misses.

Automated detection of shop names and signage on a Berlin storefront.

Challenges and Future Work

Several hurdles remain for scaling this to a city-wide level:

  • Inaccurate Geo-matching: Currently, each image is associated with only one building footprint. In reality, one photo might capture a whole row of shops, leading to data being assigned to the wrong address.
  • Linguistic Nuances: Standard STR models struggled with German special characters like 'ä' or 'ü', often misidentifying them as their base vowels.
  • Data Distribution: Flickr data is clustered around tourist "hotspots." Achieving a uniform city-wide map will require integrating other data sources like Mapillary or improving the motivation for contributors to photograph less iconic residential areas.

Future systems must combine STR with broader object detection (to recognize logos and ATMs) and potentially utilize super-resolution modules to handle low-quality, distant text regions.

Summary updated

Abstract

Crowdsourced platforms provide huge amounts of street-view images that contain valuable building information. This work addresses the challenges in applying Scene Text Recognition (STR) in crowdsourced street-view images for building attribute mapping. We use Flickr images, particularly examining texts on building facades. A Berlin Flickr dataset is created, and pre-trained STR models are used for text detection and recognition. Manual checking on a subset of STR-recognized images demonstrates high accuracy. We examined the correlation between STR results and building functions, and analysed instances where texts were recognized on residential buildings but not on commercial ones. Further investigation revealed significant challenges associated with this task, including small text regions in street-view images, the absence of ground truth labels, and mismatches in buildings in Flickr images and building footprints in OpenStreetMap (OSM). To develop city-wide mapping beyond urban hotspot locations, we suggest differentiating the scenarios where STR proves effective while developing appropriate algorithms or bringing in additional data for handling other cases. Furthermore, interdisciplinary collaboration should be undertaken to understand the motivation behind building photography and labeling. The STR-on-Flickr results are publicly available at https://github.com/ya0-sun/STR-Berlin.
BibTeX
@inproceedings{sun2023towards,
  author = {Yao Sun and Anna Kruspe and Liqiu Meng and Yajie Tian and Eike Jens Hoffmann and Stefan Auer and Xinhua Zhu},
  title = {{TOWARDS LARGE-SCALE BUILDING ATTRIBUTE MAPPING USING CROWDSOURCED IMAGES: SCENE TEXT RECOGNITION ON FLICKR AND PROBLEMS TO BE SOLVED}},
  booktitle = {˜The},
  year = {2023},
  doi = {10.5194/isprs-archives-xlviii-1-w2-2023-225-2023},
}
Figure