Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images

Yi Zhu, Xueqing Deng, Shawn Newsam

Traditional city mapping has a "view from above" problem. While satellite and aerial imagery are excellent for identifying land cover—distinguishing a forest from a parking lot—they struggle with land use. From 30,000 feet, a building housing a trendy bistro looks identical to one housing a hardware store. This is known as the "semantic gap."

In their paper, "Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images," researchers from UC Merced propose a shift in perspective. By leveraging the millions of geotagged photos we upload to the web, they’ve built a system that "sees" the city as we do, enabling fine-grained classification across 45 distinct categories.

TL;DR

The researchers developed a framework to map city land use by analyzing ground-level Flickr images using a two-stream (object and scene) convolutional neural network. By overcoming the inherent noisiness of social media data, they achieved a 29% recall rate across 45 complex land-use classes in San Francisco, establishing a new baseline for automated urban mapping.

What Problem Does This Paper Address?

The core challenge is the lack of up-to-date, detailed land-use maps. Traditional maps rely on manual surveys that are updated only every 5–10 years. While remote sensing (satellites) helps, it cannot see "inside" the functional life of a building.

To solve this, the authors turn to geotagged social multimedia. However, using web images introduces three major hurdles:

  1. Lack of Ground Truth: There is no "gold standard" map to train or evaluate these models at a fine-grained level.
  2. Scale and Labels: Manually labeling millions of web photos is impossible.
  3. Data Noise: Web images are messy. They have inaccurate geotags, varying quality, and uneven spatial distribution (lots of photos of landmarks, few of laundromats).

Methods

The authors built an end-to-end pipeline to turn noisy web photos into a structured map:

  • Two-Stream CNN: Recognizing a "pharmacy" requires seeing both objects (bottles, shelves) and the scene (a retail interior). The architecture uses two ResNet101 streams: one pre-trained on ImageNet (object-centric) and one on Places365 (scene-centric). The results are fused via late fusion.
  • Online Adaptive Training: To handle noisy labels from keyword searches, the team implemented a "cleaning" strategy during fine-tuning. The model calculates a probability score for each sample; if the model is highly confused by an image (nearly uniform prediction scores), it discards that sample to prevent the "noise" from corrupting the weights.
  • Geo-filtering with Shapefiles: Instead of treating the city as a grid, they used GIS shapefiles (polygonal outlines of land parcels). They filtered Flickr images to ensure they fell within 5 meters of a known parcel boundary, significantly reducing spatial noise.

Datasets

  • Training Set: A massive collection of over 2 million images scraped from Google and Flickr using keyword searches for the 45 target classes.
  • Mapping (Test) Set: 96,382 geotagged Flickr images from San Francisco (2016), filtered down to ~58,000 via geo-filtering.
  • The San Francisco Ground Truth Map: A major contribution of the paper. The authors combined the Land Based Classification Standards (LBCS) with the Google Places API to create a benchmark map for San Francisco with a three-level hierarchy (5 top classes, 16 middle, 45 fine-grained).

Key Insights and Ideas

  • Objects vs. Scenes: The "Object" stream generally outperformed the "Scene" stream. For example, a "bicycle store" is easily identified by the presence of bikes, even if the "scene" looks like a generic retail space. However, combining both streams provided a significant boost (nearly 3% accuracy gain).
  • The "Bicycle Store" Paradox: Some classes have high image-level accuracy but low mapping recall. The model is great at identifying a bike shop if it sees one, but because bike shops are small and people rarely take photos inside them, they are hard to "map" at scale.
  • Beneficial Noise: The authors found that a certain level of label noise actually helps deep neural networks generalize better, provided the dataset is large enough and a strategy like Online Adaptive Training is used to prune the most egregious errors.

Key Results

  • Image Classification: The final two-stream model achieved 49.54% accuracy on the 45-way classification task, a significant jump over the SIFT baseline (29.16%).
  • City Mapping: At the individual land parcel level, the framework achieved over 29% recall. While 29% might sound low in standard CV tasks, it represents a powerful baseline for a 45-class problem using completely uncurated, noisy social media data.
  • Hierarchy Matters: Accuracy scales with granularity. The model hits 75.6% accuracy when asked to distinguish between 5 broad categories (e.g., Residential vs. Industrial), but drops as the "semantic gap" narrows at the 45-class level.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Spatial Sparsity: Even with 400 million Flickr images, many parts of a city remain "dark" because they aren't "Instagrammable" (e.g., industrial zones or private government offices).
  • Privacy and Intent: People don't take photos in banks or courthouses as often as in parks, leading to data imbalances.
  • Human-in-the-loop: The authors suggest that future systems could use "active learning," where the model asks humans to label the most ambiguous cases to improve accuracy.
  • Multi-modal Fusion: Integrating audio, video, or even text from tweets could help fill the gaps where images are missing or ambiguous.

Summary updated

Abstract

Multimedia researchers have exploited large collections of community-contributed geo-referenced images to better understand a particular image, such as its subject matter or where it was taken, as well as to better understand a geographic location, such as the most visited tourist spots in a city or what the local cuisine is like. The goal of this paper is to better understand location. In particular, we use geo-referenced image collections to better understand what occurs in different parts of a city at fine spatial and activity class scales. This problem is known as land use mapping in the geographical sciences. We propose a novel framework to perform fine-grained land use mapping at the city scale using ground-level images. Mapping land use is considerably more difficult than mapping land cover and is generally not possible using overhead imagery as it requires close-up views and seeing inside buildings. We postulate that the growing collections of geo-referenced, ground-level images suggest an alternate approach to this geographic knowledge discovery problem. We develop a general framework that uses Flickr images to map 45 different land-use classes for the city of San Francisco, CA, USA. Individual images are classified using a novel convolutional neural network containing two streams: one for recognizing objects and another for recognizing scenes. This network is trained in an end-to-end manner directly on the labeled training images. We propose several novel strategies to overcome the noisiness of our user-generated data including search-based training set augmentation and online adaptive training. We derive a ground truth map of San Francisco in order to evaluate our method. We demonstrate the effectiveness of our approach through geovisualization and quantitative analysis. Our framework achieves over 29% recall at the individual land parcel level that represents a strong baseline for the challenging 45-way land use classification problem, especially given the noisiness of the image data.
BibTeX
@article{zhu2019fine,
  author = {Yi Zhu and Xueqing Deng and Shawn Newsam},
  title = {{Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images}},
  journal = {IEEE Transactions on Multimedia},
  year = {2019},
  doi = {10.1109/tmm.2019.2891999},
}

Loading related content...

Figure