Automatic Classification of Photos by Tourist Attractions Using Deep Learning Model and Image Feature Vector Clustering

Jiyeon Kim, Youngok KangISPRS Int. J. Geo Inf 2022 · DOI

TL;DR

This paper introduces a novel method for automatically classifying tourist photos by attraction using deep learning and image clustering. By leveraging pre-trained models and unsupervised clustering, it overcomes the limitations of manual categorization and generic datasets, enabling more nuanced analysis of tourist preferences and destination images.

What Problem Does This Paper Address?

The proliferation of social media has made user-generated content (UGC), particularly photos, a crucial factor in shaping tourism destination images (TDI) and influencing tourist decisions. While deep learning has shown promise in analyzing these photos, existing methods often struggle with classifying unique attractions due to reliance on generic datasets (like ImageNet or Places365) or require extensive manual effort to create custom datasets and categories. This gap makes it difficult to extract specific visual elements that make individual tourist attractions appealing.

Methods

The proposed method involves a four-step process:

  1. Data Collection: Photos from foreign tourists' reviews on TripAdvisor were collected.
  2. Image Embedding: A VGG16 network, pre-trained on Places365, was used to convert each image into a 512-dimensional feature vector.
  3. Dimension Reduction and Clustering: These high-dimensional vectors were reduced to 2D using t-SNE (t-Distributed Stochastic Neighbor Embedding) for visualization and analysis. Subsequently, HDBSCAN (Hierarchical Clustering and Density-Based Spatial Clustering of Applications with Noise) was applied to these 2D embeddings to automatically identify clusters, which were then treated as image categories for specific attractions.
  4. Noise Removal and Classification: A Siamese Network, trained on a custom dataset derived from the identified clusters, was employed to remove outlier (noise) photos within clusters and classify the remaining photos into their respective categories. Triplet loss with semi-hard mining was used as the cost function for training the Siamese network.

Datasets

  • TripAdvisor: Photos were collected from reviews of tourist attractions on TripAdvisor. Specifically, data from 'Gyeongbokgung Palace' and 'Insadong' in Seoul, South Korea, were used.
  • Places365: A pre-trained VGG16 model on the Places365 dataset was utilized for image embedding. Places365 is a large-scale dataset containing 365 categories of scenes.

Key Insights and Ideas

  • Data-Driven Category Generation: The core innovation is using unsupervised clustering (HDBSCAN) on image embeddings to automatically generate relevant categories for tourist attractions, eliminating the need for manual category definition.
  • Flexibility for Specific Attractions: This approach allows for flexible category extraction tailored to the unique visual characteristics of each tourist destination, unlike generic classification datasets.
  • Siamese Network for Refinement: The Siamese network effectively acts as a post-clustering refinement step, removing noise and improving the accuracy of classification within the automatically generated categories, especially useful when dealing with potentially noisy cluster boundaries.
  • Addressing Data Scarcity: The Siamese network's ability to perform well with smaller datasets is highlighted as a significant advantage for analyzing specific tourist attractions where large, labeled datasets might be scarce.

Key Results

  • Gyeongbokgung Palace: 10 categories were automatically generated from 8188 photos. The Siamese network achieved a TPR of 0.928 and an FPR of 0.045.
  • Insadong: 4 categories were generated from 3695 photos. The Siamese network achieved a TPR of 0.90 and an FPR of 0.057.
  • The method successfully identified visual elements that are attractive to tourists, contributing to the formation of destination images.
  • The study demonstrated that the proposed method can improve classification performance even with a relatively small dataset.

Limitations and Future Work

  • Comparison with Other SNS: The authors suggest comparing the proposed method's results with photos from other social media platforms (beyond TripAdvisor) that might offer a wider variety of images for the same area.
  • Comparison with Existing Methods: A comparison with categories created by existing methods that use predefined photo categories (e.g., Places365, ImageNet) is recommended to further validate the proposed approach.
  • Data Diversity: While TripAdvisor data is useful, it might have limitations in terms of image diversity compared to other platforms. Future work could explore integrating data from multiple sources.

Summary updated

BibTeX
@article{kim2022automatic,
  author = {Jiyeon Kim and Youngok Kang},
  title = {{Automatic Classification of Photos by Tourist Attractions Using Deep Learning Model and Image Feature Vector Clustering}},
  journal = {ISPRS Int. J. Geo Inf.},
  year = {2022},
  doi = {10.3390/ijgi11040245},
}
Figure