Scalable Urban Data Collection from the Web

Rijurekha Sen, Daniele Quercia, Carmen Vaca, Krishna P. GummadiAAAI 2021 · DOI

Abstract

Easy access to different necessities of daily life makes a city more livable. This has motivated urban planning researchers to quantify urban accessibility from official city data. However, due to the manual nature of data collection, these earlier survey based analyses were limited in scope and scalability, and mostly offered insights on cities of developed countries like the UK and the USA. Using Google Places data that is crowd-sourced around the world, this paper gathers walkability information for twenty-five cities across five continents. We detail the collection methodology of this unprecedented dataset and show useful applications of this data in urban analysis: e.g., how different areas within a city compare against each other in terms of accessibility and which areas in a city would benefit the most from the least intervention.

Introduction

Urban planners have long recognized that "livability" is fundamentally tied to how easily residents can access daily necessities—such as groceries, schools, and healthcare—on foot. Traditionally, quantifying this required manual surveys and official city records, which are labor-intensive, slow to update, and often unavailable in developing nations. While web-based services like Walk Score exist, they are often proprietary or expensive to scale.

This research addresses the "data gap" in urban planning by proposing a reproducible, global, and cost-effective method to measure urban walkability. By leveraging Google Maps APIs, we have developed a scalable, automated approach to create high-resolution "accessibility maps" that identify neighborhood-level service gaps, providing a blueprint for data-driven urban interventions.

Methodology

The proposed methodology utilizes a three-step "crawling" process designed for efficiency and reproducibility:

  1. Grid Decomposition: The target city is divided into a grid of 200m x 200m squares. The centroid of each square serves as the origin point for accessibility analysis.
  2. Facility Discovery: Using the Google Places API, the system queries for 30 distinct categories of facilities (e.g., pharmacies, schools, transit stations) near each centroid. To optimize API usage, similar categories are grouped into logical "blocks."
  3. Distance Mapping: Once the nearest facility in each category is identified, the Google Distance Matrix API calculates the actual walking distance and estimated walking time from the grid center to that facility.
Figure 2: Grid decomposition of London into 200m x 200m squares for data collection.

The final output is an area-by-category distance matrix, where rows represent specific 200m blocks and columns represent the 30 facility types.

Datasets

The study introduces an unprecedented dataset covering 25 cities across five continents, categorized by development status:

  • Industrialized Europe: Barcelona, Berlin, London, Milan, Paris, Rome.
  • Industrialized North America: Chicago, New York, San Francisco, Seattle, Toronto, Washington D.C.
  • Industrialized Asia: Beijing, Singapore, Tokyo.
  • Developing Regions: Bengaluru, Buenos Aires, Delhi, Jakarta, Kuala Lumpur, Mexico City, Moscow, Mumbai, Nairobi, Rio de Janeiro.

This diversity allows for a global perspective, moving beyond the Western-centric focus typical of previous urban studies.

Analysis and Insights

The core insight of this work is that crowdsourced data from platforms like Google Maps serves as a robust proxy for official urban data, particularly in regions where government data is non-existent.

Instead of a single, reductive "walkability score," we employ K-means clustering to group neighborhoods with similar accessibility profiles. This allows planners to identify "poor" clusters (areas with long distances to many facilities) and compare them to "rich" clusters. Such comparisons reveal specific, targeted interventions. For instance, if a "poor" cluster lacks only a pharmacy to match the profile of a "rich" cluster, the intervention is clear and low-cost.

Results

The methodology successfully mapped cities at a 200m resolution, visualizing accessibility "deserts" versus "oases." Validation through K-means clustering (tested at K=4 and K=8) effectively distinguished between high-density city centers and sparse peripheral areas.

Figure 4: K-means clustering of accessibility in London with four (left) and eight (right) clusters.

The method proved capable of processing diverse urban layouts, from the dense, historic grids of Europe to the sprawling megacities of the developing world, using a manageable number of API calls.

Conclusion and Future Work

While the current model provides a powerful baseline, it assumes static walking speeds and does not yet account for real-time traffic, sidewalk quality, or topography. Future research will utilize this dataset to analyze urban structures (mono-centric vs. poly-centric) and conduct deeper comparative studies between European and American urban designs to quantify the "accessibility gap" in urban theory.

Summary updated

Abstract

Easy access to different necessities of daily life makes a city more livable. This has motivated urban planning researchers to quantify urban accessibility from official city data. However, due to the manual nature of data collection, these earlier survey based analyses were limited in scope and scalability, and mostly offered insights on cities of developed countries like the UK and the USA. Using Google Places data that is crowd-sourced around the world, this paper gathers walkability information for twenty-five cities across five continents. We detail the collection methodology of this unprecedented dataset and show useful applications of this data in urban analysis: e.g., how different areas within a city compare against each other in terms of accessibility and which areas in a city would benefit the most from the least intervention.
BibTeX
@inproceedings{sen2021scalable,
  author = {Rijurekha Sen and Daniele Quercia and Carmen Vaca and Krishna P. Gummadi},
  title = {{Scalable Urban Data Collection from the Web}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/icwsm.v10i1.14808},
}
Figure