Vis2Rec: A Large-Scale Visual Dataset for Visit Recommendation

Michaël Soumm, Adrian Popescu, Bertrand DelezoideWACV 2023

TL;DR

This paper introduces Vis2Rec, a novel, large-scale dataset of over 7 million Flickr photos from 14,600 users, designed to enable visual-based Point of Interest (POI) recommendation. By leveraging deep learning for visual matching and addressing data privacy concerns, Vis2Rec aims to overcome the limitations of existing datasets and pave the way for more personalized and richer travel recommendations.

What Problem Does This Paper Address?

Recommending personalized travel experiences is a significant challenge. Traditional methods often rely on explicit user check-ins or limited metadata, which don't fully capture a user's preferences. Tourists, however, extensively document their trips through photos, which contain a wealth of implicit information about their interests and visited locations. While visual content has been explored for recommendations, there's a lack of large-scale, publicly available datasets that specifically link user photos to POIs for recommendation tasks. This gap hinders the development and benchmarking of advanced, image-based recommendation algorithms. Furthermore, existing datasets often have limitations regarding data provenance, rights, and privacy, making them difficult to use or share.

Methods

The core of this work lies in constructing the Vis2Rec dataset and demonstrating its utility. The process involves several key steps:

  • Initial Data Collection: Leveraging the Flickr API, the authors collected photos shared under Creative Commons licenses. This ensures the data is legally distributable.
  • POI Identification: To identify POIs within user photos, they utilized the Google Landmarks v2 (GLv2) dataset and a deep learning model (DELG) for visual matching. This approach avoids the need for explicit user annotations.
  • Domain-Related Data Selection: Photos were filtered to represent actual tourist visits by identifying "visit days" based on coarse POI predictions and tag analysis. Geolocation data was used to refine POI identification within a radius.
  • Visual Matching: A two-step visual matching process using DELG descriptors was employed. First, a global embedding selects candidate reference images, followed by geometric verification using local descriptors to refine matches and attribute POIs. This process was further refined by filtering out geographically inconsistent matches.
  • Dataset Partitioning and Annotation: The dataset was split into training, validation, and test sets. For the test and validation sets, a manual annotation process was conducted to establish ground truth visits, ensuring high accuracy for evaluation.
  • Privacy Preservation: To address privacy concerns, faces in images were de-identified, and only freely distributable content was used.

Datasets

  • Vis2Rec (New Benchmark): A large-scale dataset comprising:
  • 7,158,454 total photos
    • 14,600 visiting users
    • 36,111 unique POIs
    • 421,065 unique POI visits
    • Data collected from Flickr, filtered for Creative Commons licenses.
  • Google Landmarks v2 (GLv2): Used as an auxiliary dataset for visual POI recognition.
  • Comparison Datasets (Table 1): The paper compares Vis2Rec to existing recommender system datasets like Amazon clothing, MovieLens, Netflix, Foursquare, and Gowalla, highlighting Vis2Rec's unique focus on user images for travel recommendations.

Key Insights and Ideas

  • Visual Data as Rich User Profiles: The paper strongly advocates for the power of visual data (photos) as a rich source for building user profiles in recommendation systems, moving beyond traditional explicit feedback.
  • Scalable Visual POI Recognition: The successful application of deep learning models like DELG for large-scale visual POI recognition is crucial for creating such datasets without manual annotation.
  • Privacy-Preserving Data Collection: The emphasis on using Creative Commons licensed content and de-identifying faces demonstrates a commitment to ethical data handling, which is vital for public datasets.
  • "Visit Day" Concept: The novel approach of defining "visit days" based on photo clusters and tags helps to structure temporal user activity and identify meaningful sequences of visits.
  • Long-Tail Distribution of POIs: The findings on POI distribution align with real-world tourist trends, showing a concentration in popular cities and a long tail of less frequently visited locations, which presents a challenge for recommendation systems.

Key Results

  • Dataset Size: Vis2Rec is one of the largest publicly available datasets for visual POI recommendation, containing over 7 million photos and data for 14,600 users.
  • Benchmarking Performance: The paper benchmarks several recommendation algorithms on Vis2Rec. The RecVAE model achieved the best performance across metrics like Recall@N and NDGC@N, outperforming simpler methods like MostPop and MF.
  • Ablation Study: Removing 50% of identified POIs had the most significant negative impact (up to 7 percentage points reduction in RecVAE performance), highlighting the importance of comprehensive POI identification. Ablating user profiles had a less pronounced effect, suggesting the dataset's size helps mitigate the impact of reduced user data.
  • Qualitative Visual Matching: The DELG model demonstrated good qualitative results in identifying POIs across various conditions, though errors were observed for visually similar objects or signs.

Limitations and Future Work

  • Data Representativeness: The dataset primarily contains photos taken during vacations, potentially not reflecting everyday life or all types of tourist activities.
  • Visual Matching Errors: While DELG performs well, errors in POI identification can still occur, impacting profile accuracy.
  • Future Work:
    • Improved Visual Matching: Incorporating more advanced visual matching algorithms.
  • Leveraging Additional Cues: Utilizing other information within photos (e.g., semantic content beyond POIs) to create richer user profiles.
  • Geo-Temporal Data: Exploring more sophisticated use of geo-temporal data for recommendations.
  • Testing New Algorithms: Evaluating newer recommendation algorithms on Vis2Rec.
  • On-Device Processing: The authors suggest that the proposed use-case could be achieved on-device, enhancing privacy.

Summary updated

BibTeX
@inproceedings{soumm2023vis,
  author = {Michaël Soumm and Adrian Popescu and Bertrand Delezoide},
  title = {{Vis2Rec: A Large-Scale Visual Dataset for Visit Recommendation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure