Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images

Yi Zhu, Xueqing Deng, Shawn Newsam

Abstract

We perform fine-grained land use mapping at the city scale using ground-level images. Mapping land use is considerably more difficult than mapping land cover and is generally not possible using overhead imagery as it requires close-up views and seeing inside buildings. We postulate that the growing collections of georeferenced, ground-level images suggest an alternate approach to this geographic knowledge discovery problem. We develop a general framework that uses Flickr images to map 45 different land-use classes for the City of San Francisco. Individual images are classified using a novel convolutional neural network containing two streams, one for recognizing objects and another for recognizing scenes. This network is trained in an end-to-end manner directly on the labeled training images. We propose several strategies to overcome the noisiness of our user-generated data including search-based training set augmentation and online adaptive training. We derive a ground truth map of San Francisco in order to evaluate our method. We demonstrate the effectiveness of our approach through geo-visualization and quantitative analysis. Our framework achieves over 29% recall at the individual land parcel level which represents a strong baseline for the challenging 45-way land use classification problem especially given the noisiness of the image data.

Introduction: The Semantic Gap

Traditional urban mapping faces a "view from above" problem. While satellite and aerial imagery are excellent for identifying land cover—distinguishing a forest from a parking lot—they struggle with land use. From 30,000 feet, a building housing a trendy bistro looks identical to one housing a hardware store. This is known as the "semantic gap."

In their research, UC Merced scientists propose a shift in perspective. By leveraging the millions of geotagged photos uploaded to social media, they have developed a system that "sees" the city from the ground up, enabling fine-grained classification across 45 distinct categories.

Methodology

The researchers built an end-to-end pipeline to transform noisy web photos into structured geographic data:

  • Two-Stream CNN: Recognizing a specific land use, like a "pharmacy," requires identifying both objects (bottles, shelves) and the scene (a retail interior). The architecture uses two ResNet101 streams: one pre-trained on ImageNet (object-centric) and one on Places365 (scene-centric), with results fused via late fusion.
  • Online Adaptive Training: To handle noisy labels from keyword searches, the model calculates a probability score for each sample. If the model is highly confused by an image, it discards that sample during fine-tuning to prevent noise from corrupting the weights.
  • Geo-filtering with Shapefiles: Instead of a simple grid, the system uses GIS shapefiles (polygonal outlines of land parcels). Images are filtered to ensure they fall within 5 meters of a known parcel boundary, significantly reducing spatial noise.

Datasets and Ground Truth

A major contribution of this work is the creation of a ground truth map for San Francisco. The authors combined the Land Based Classification Standards (LBCS) with the Google Places API to create a benchmark map with a three-level hierarchy.

Spatial distribution of labeled vs. unlabeled land parcels in San Francisco.

The training set utilized over 2 million images scraped from Google and Flickr, while the mapping (test) set consisted of approximately 58,000 filtered Flickr images from San Francisco.

Key Results and Insights

The final two-stream model achieved 49.54% accuracy on the 45-way image classification task. At the individual land parcel level, the framework achieved over 29% recall. While 29% might seem low for standard computer vision tasks, it represents a powerful baseline for a 45-class problem using uncurated, noisy social media data.

Fine-grained land use classification results across 45 categories in a San Francisco neighborhood.

Key Insights:

  • Objects vs. Scenes: The "Object" stream generally outperformed the "Scene" stream. For instance, a "bicycle store" is easily identified by the presence of bikes, even if the scene looks like a generic retail space. Combining both streams provided a significant 3% accuracy gain.
  • Spatial Sparsity: Many parts of a city remain "dark" because they aren't "Instagrammable" (e.g., industrial zones or private offices). This spatial sparsity remains a challenge for social-media-based mapping.
  • Beneficial Noise: Interestingly, a certain level of label noise helped the neural networks generalize better, provided the dataset was large enough and adaptive training was used to prune egregious errors.

Conclusion

By bridging the semantic gap with ground-level imagery, this framework offers a scalable approach to urban knowledge discovery. Future work may integrate multi-modal data—such as audio or text—to fill the gaps where images are missing or ambiguous, further refining our digital understanding of the urban pulse.

Summary updated

Abstract

We perform fine-grained land use mapping at the city scale using ground-level images. Mapping land use is considerably more difficult than mapping land cover and is generally not possible using overhead imagery as it requires close-up views and seeing inside buildings. We postulate that the growing collections of georeferenced, ground-level images suggest an alternate approach to this geographic knowledge discovery problem. We develop a general framework that uses Flickr images to map 45 different land-use classes for the City of San Francisco. Individual images are classified using a novel convolutional neural network containing two streams, one for recognizing objects and another for recognizing scenes. This network is trained in an end-to-end manner directly on the labeled training images. We propose several strategies to overcome the noisiness of our user-generated data including search-based training set augmentation and online adaptive training. We derive a ground truth map of San Francisco in order to evaluate our method. We demonstrate the effectiveness of our approach through geo-visualization and quantitative analysis. Our framework achieves over 29% recall at the individual land parcel level which represents a strong baseline for the challenging 45-way land use classification problem especially given the noisiness of the image data.
BibTeX
@article{zhu2019fine,
  author = {Yi Zhu and Xueqing Deng and Shawn Newsam},
  title = {{Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images}},
  journal = {IEEE Transactions on Multimedia},
  year = {2019},
  doi = {10.1109/tmm.2019.2891999},
}
Figure