An Analysis of Human-Centered Geolocation

Kaili Wang, Yu-Hui Huang, José Oramas M., Luc Van Gool, Tinne Tuytelaars

TL;DR

Researchers have demonstrated that AI can geolocate "human-centered" photos—images where a person dominates the frame—by analyzing fashion trends, physical appearance, and subtle contextual cues. Their Convolutional Neural Network (CNN) models significantly outperformed humans, proving that what we wear and how we look are surprisingly strong indicators of our geographic location.

What Problem Does This Paper Address?

Traditional image geolocation usually relies on "landmark recognition"—identifying the Eiffel Tower or a specific street corner in London. However, a massive portion of social media imagery is "human-centered," focusing on a single individual with minimal background. In these photos, the background is often blurred or generic (like a plain wall), making traditional methods fail.

The authors ask: Is it possible to geolocate an image using only the person as the primary cue? They hypothesize that cultural and climate factors create distinct "visual signatures" for different cities—think tanned skin and summer dresses in Miami versus heavy coats and scarves in Moscow.

Methods

The researchers approached the problem through three main lenses:

  1. Human Baseline: They conducted a user study where participants tried to guess the city of a photo from 12 possibilities.
  2. Automated Classification: They employed a VGG-F Convolutional Neural Network architecture. They tested three configurations:
  3. Pretrained+SVM: Using a network trained on ImageNet as a fixed feature extractor.
  4. Finetuned: Training the network specifically on geolocation datasets.
  5. Finetuned+SVM: Using the finetuned network's internal activations to train a separate classifier.
  6. Feature Inspection: To ensure the model wasn't just "cheating" by looking at the background, they used Faster R-CNN to detect and crop only the person (human-based feature pooling). They also used upsampled activation maps to visualize exactly which parts of the image the network was "looking at" when making a prediction.

Datasets

The study utilized three primary datasets covering 12 major global cities (e.g., LA, London, Tokyo, Paris):

  • Fashion 144k: A dataset derived from the fashion site Chictopia.
  • Self-Collected Chictopia/Pinterest: Two new datasets curated by the authors to represent different photography styles ("planned" poses vs. "instant" snapshots).
  • Large-Scale Expansion: A subset of Fashion 144k covering 164 cities (70,000 images) to test if the findings hold at scale.

Key Insights and Ideas

  • AI vs. Human Intuition: Humans are surprisingly bad at this. Participants often relied on stereotypes (e.g., looking for snow to identify a "cold" city), which led to errors if the photo didn't match the trope. The AI, having "seen" thousands of examples, picked up on more nuanced, statistically relevant patterns.
  • The "Black Box" Unveiled: By inspecting activation maps, the researchers found the network focused on specific regions: faces and arms (to detect skin tone/tanning) and clothing types (sleeves vs. tank tops).
  • Context Still Matters: While the "human-only" crops performed remarkably well, the models performed 4–6% better when they had access to the full image. This suggests that while fashion is the primary signal, "micro-contexts" like wall textures or local vegetation provide a helpful boost.
  • Commercial Potential: Beyond academic interest, this has massive implications for retail. Multinational brands could use these insights to understand regional fashion trends or provide geography-based product recommendations.

Key Results

  • Human Performance: In a 12-city choice task, humans achieved only 11.94% accuracy (barely above the 8.33% chance level). Even when given "training" images, they only reached ~21%.
  • Model Performance: The "Finetuned" CNN achieved 40.75% accuracy on the same 12-city task—nearly double the best human performance.
  • Human-Only Accuracy: When the background was completely removed, the model still achieved 35% accuracy, proving that human-centric features are indeed highly informative.
  • Scalability: In the massive 164-city experiment, the model maintained a 38.26% Mean Class Accuracy (mCA), showing the approach remains robust even as the number of possible locations increases.

Limitations and Future Work

The authors acknowledge several hurdles:

  • City Similarity: The model (and humans) frequently confused "visually similar" cities, such as Paris and London, or Miami and Los Angeles.
  • The "Training" Gap: A major reason for the AI's success is its ability to process thousands of images, whereas humans in the study had limited exposure to the "visual language" of each city.
  • Future Directions: The researchers suggest that integrating clothing parsing (specifically identifying "jacket" vs. "shirt") could further refine the feature pooling. They also want to investigate how indoor vs. outdoor settings affect the reliability of these fashion-based cues.

Summary updated

Abstract

Online social networks contain a constantly increasing amount of images - most of them focusing on people. Due to cultural and climate factors, fashion trends and physical appearance of individuals differ from city to city. In this paper we investigate to what extent such cues can be exploited in order to infer the geographic location, i.e. the city, where a picture was taken. We conduct a user study, as well as an evaluation of automatic methods based on convolutional neural networks. Experiments on the Fashion 144k and a Pinterest-based dataset show that the automatic methods succeed at this task to a reasonable extent. As a matter of fact, our empirical results suggest that automatic methods can surpass human performance by a large margin. Further inspection of the trained models shows that human-centered characteristics, like clothing style, physical features, and accessories, are informative for the task at hand. Moreover, it reveals that also contextual features, e.g. wall type, natural environment, etc., are taken into account by the automatic methods.
BibTeX
@inproceedings{wang2018analysis,
  author = {Kaili Wang and Yu-Hui Huang and José Oramas M. and Luc Van Gool and Tinne Tuytelaars},
  title = {{An Analysis of Human-Centered Geolocation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  doi = {10.1109/wacv.2018.00227},
}
Figure