Towards Lifestyle Understanding: Predicting Home and Vacation Locations from User's Online Photo Collections

Danning Zheng, Tianran Hu, Quanzeng You, Henry Kautz, Jiebo Luo

Abstract

Semantic place labeling has been actively studied in the past few years due to its importance in understanding human mobility and lifestyle patterns. In the last decade, the rapid growth of geotagged multimedia data from online social networks provides a valuable opportunity to predict people's POI locations from temporal, spatial and visual cues. Among the massive amount of social media data, one important type of data is the geotagged web images from image-sharing websites. In this paper, we develop a reliable photo classifier based on the Convolutional Neutral Networks to classify the photo-taking scene of real-life photos. We then present a novel approach to home location and vacation locations prediction by fusing together the visual content of photos and the spatiotemporal features of people's mobility patterns. Using a well-trained classifier, we showed that the robust fusion of visual and spatiotemporal features achieves significant accuracy improvement over each of the features alone for both home and vacation detection.

Introduction: The Semantic Challenge of Mobility

Understanding human mobility is crucial for urban planning, transportation design, and epidemiological modeling. While traditional methods rely on GPS traces or call records, these datasets are often proprietary, expensive, and difficult to scale. Social media platforms like Flickr offer a massive, public alternative. However, these data points present a "semantic" challenge: a single geographic coordinate—such as a beach—carries different meanings for different users. For a lifeguard, it is a workplace; for a tourist, it is a vacation spot. This research focuses on identifying the two primary anchors of a person's life: their home and their vacation locations.

Methodology: Fusing Pixels and Patterns

The researchers proposed a multi-source approach that integrates visual content with spatiotemporal metadata.

  1. Visual Scene Classification: A Convolutional Neural Network (CNN) was trained to distinguish "home-like" scenes (kitchens, living rooms, bedrooms) from "non-home" environments. While traditional computer vision techniques like HOG features struggle with the clutter of real-life photos, the deep learning model proved robust against variations in lighting and the presence of people or pets.
  2. Temporal Feature Extraction: The system analyzes 16 temporal features, including the "upload rate" (the percentage of a user's total photos taken in a specific month) and "active months" (the count of months per year a user visits a specific location).
  3. Spatial Analysis: For vacation detection, the model calculates the distance from the predicted home location, using a sigmoid function to normalize this spatial gap.
  4. Fusion Model: A Bayesian Network meta-classifier fuses these disparate visual and spatiotemporal features into a unified prediction engine.

Key Insights: The Rhythm of Home Life

A core finding of the study is that visual and metadata features have complementary strengths. For instance, a restaurant might visually resemble a home due to the presence of food and people, but temporal data revealing infrequent visits can correct a false positive. Conversely, an office might share a temporal pattern with a home (high frequency of visits), but the visual classifier can identify professional furniture to filter it out.

Photo-taking rates by month for home vs. non-home locations, highlighting the December spike.

One of the most distinctive temporal signatures discovered was the "December Spike." As shown in the data, nearly 20% of all "home" photos are taken in December, likely reflecting holiday gatherings. Additionally, the researchers introduced "photo efficiency" (the ratio of photos taken to active days). Vacations are typically characterized by high efficiency: intense bursts of photography over a very short duration.

Key Results

The fusion of visual and spatiotemporal data significantly outperformed models relying on single feature sets or simple frequency baselines.

Prediction accuracy across different error distances for various feature combinations.
  • Home Prediction: The fused model achieved 71% accuracy at a 70-meter resolution (roughly a city block) and 79% accuracy within 1 kilometer.
  • Vacation Prediction: The model reached an AUC of 0.854, a marked improvement over using only spatiotemporal data (0.781) or visual data (0.787).
  • Visual Impact: Interestingly, visual features alone proved more reliable for home detection than temporal features alone, indicating that the "pixels" of indoor environments are powerful indicators of residency.

Conclusion and Future Directions

While the system is highly effective, "messy" real-life photos—such as professional studios that mimic home environments—remain a challenge. Furthermore, vacation detection currently requires a minimum threshold of 20 photos to ensure reliability. Future work will aim to expand these categories to include "work" locations and leverage these lifestyle patterns to build personalized vacation recommendation systems.

Summary updated

Abstract

Semantic place labeling has been actively studied in the past few years due to its importance in understanding human mobility and lifestyle patterns. In the last decade, the rapid growth of geotagged multimedia data from online social networks provides a valuable opportunity to predict people's POI locations from temporal, spatial and visual cues. Among the massive amount of social media data, one important type of data is the geotagged web images from image-sharing websites. In this paper, we develop a reliable photo classifier based on the Convolutional Neutral Networks to classify the photo-taking scene of real-life photos. We then present a novel approach to home location and vacation locations prediction by fusing together the visual content of photos and the spatiotemporal features of people's mobility patterns. Using a well-trained classifier, we showed that the robust fusion of visual and spatiotemporal features achieves significant accuracy improvement over each of the features alone for both home and vacation detection.
BibTeX
@inproceedings{zheng2021towards,
  author = {Danning Zheng and Tianran Hu and Quanzeng You and Henry Kautz and Jiebo Luo},
  title = {{Towards Lifestyle Understanding: Predicting Home and Vacation Locations from User's Online Photo Collections}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/icwsm.v9i1.14591},
}
Figure