TL;DR
This paper tackles the challenge of understanding tourists' perception of urban environments by analyzing geotagged photos. They developed a specialized deep learning model, fine-tuned using transfer learning, to accurately classify tourist photos based on local characteristics, achieving high accuracy in identifying tourist attractions and urban imagery in Seoul.
What Problem Does This Paper Address?
Understanding how tourists perceive and experience urban environments is crucial for urban planning, tourism management, and marketing. Traditional methods of analyzing tourist perceptions often rely on surveys or manual analysis of limited photo collections, which are time-consuming and may not capture the nuances of visual experiences. While deep learning, particularly Convolutional Neural Networks (CNNs), has shown promise in image analysis, existing models trained on general datasets (like ImageNet or Places365) struggle to accurately capture the unique cultural and landscape characteristics of specific regions. This leads to misclassifications and an incomplete understanding of tourist-generated visual content. The authors aim to bridge this gap by developing a robust method for classifying tourist photos that is sensitive to local characteristics, thereby providing a more accurate representation of tourists' urban imagery.
Methods
The core of this research lies in the application and adaptation of transfer learning with a deep learning model, specifically the Inception-v3 architecture.
Here's a breakdown of their approach:
- Transfer Learning: Instead of training a CNN from scratch, they leverage a pre-trained Inception-v3 model (trained on the large ImageNet dataset). This allows them to benefit from the model's existing ability to recognize general visual features.
- Fine-tuning: The pre-trained Inception-v3 model was then "fine-tuned" for the specific task of classifying tourist photos. This involved:
- Custom Tourist Photo Classification: A novel classification system was developed, comprising 75 distinct scenes and 13 broader categories, with a significant portion (47%) designed to capture strong local characteristics of Korean tourism. This was informed by literature review, expert advice, and manual analysis of a subset of photos.
- Re-training the Model: The final layer of the Inception-v3 model was modified to output predictions for the 75 custom scenes. Crucially, all weights of the feature extraction part were updated, not just the classifier, to better adapt to the specific features of tourist photos.
- Data Augmentation: To combat overfitting and improve model robustness, various data augmentation techniques were employed, including zooming, rotation, brightness adjustment, horizontal flip, and width shift. This artificially increased the size and diversity of the training dataset.
- Semi-supervised Labeling: To efficiently label a large dataset, a semi-supervised approach was used. A model was initially trained on a small set of manually labeled data, then applied to label a larger dataset automatically. Human eyes were used to verify and correct these automatic labels.
- Spatial Analysis: After classification, Kernel Density Estimation (KDE) and DBSCAN were used to analyze the spatial distribution of tourist photos, identifying popular areas and understanding how different categories of photos are clustered.
Datasets
- Flickr Data: The primary dataset consisted of geotagged photos collected from Flickr.
- Initial Crawl: 284,094 photos from 5609 users were initially collected.
- Tourist Identification: A rigorous process was used to identify tourists from residents, resulting in a dataset of 168,216 photos from 3259 identified tourists.
- Training/Validation/Test Split: The data was split into 60% for training, 20% for validation, and 20% for testing.
- ImageNet: Used for the initial pre-training of the Inception-v3 model.
- Places365: Mentioned as a benchmark for comparison, but not directly used for training their custom model.
The paper introduces a newly curated dataset and classification scheme tailored for tourist photos, emphasizing local characteristics, which differs from general scene classification datasets.
Key Insights and Ideas
- Local Characteristics Matter: The most significant insight is that generic scene classification models (like those trained on ImageNet or Places365) perform poorly when applied to specific regional tourism contexts. Accurately capturing tourists' urban imagery requires a classification system that explicitly accounts for local landmarks, cultural elements, and unique experiences.
- Transfer Learning is Powerful, but Needs Customization: While transfer learning provides a strong foundation, simply applying a pre-trained model is insufficient. Fine-tuning all weights of the network, not just the classifier, is crucial for adapting to the specific nuances of tourist-generated visual content.
- Data Augmentation is Key for Robustness: The study demonstrates that data augmentation significantly improves classification accuracy, highlighting its importance when working with potentially limited or imbalanced datasets.
- "Difficult to Classify" Category is Valuable: Including a "difficult to classify" category in their scheme is a practical approach to handle ambiguous or low-quality images, preventing them from negatively impacting the model's performance on clearer examples.
- Seoul as a Microcosm: Seoul, with its blend of modern and traditional elements, serves as an excellent case study to demonstrate how tourists perceive a city that offers diverse attractions.
Key Results
- High Classification Accuracy: The final fine-tuned Inception-v3 model achieved:
-
Top 1 Accuracy: 85.77%
- Top 5 Accuracy: 95.69%
- F1-score: 0.8485 These results are comparable to, and in some cases exceed, the performance of the original Inception-v3 model on ImageNet (82.7% Top 1 accuracy).
-
Spatial Concentration: Analysis of tourist photos in Seoul revealed that photos are heavily concentrated in the downtown area, with specific hotspots identified for different categories (e.g., Myeong-dong for shopping, Gyeongbokgung Palace for traditional architecture).
- Tourist Preferences: Tourists in Seoul tend to photograph selfies, food, palaces, conference venues, and buildings. They are drawn to a mix of modern features, traditional elements, and unique cultural experiences.
- Scene-Specific Performance: The model performed best on scenes that were clearly defined or highly distinct (e.g., "bike," "cat," "eaves," "airplane," "food"), while scenes with more varied objects or less distinct features (e.g., "amusement park," "lantern and altar") showed lower performance.
Limitations and Future Work
- Limitations:
- Misclassification of Korean Characteristics: The study highlights that even with their custom classification, some Korean-specific scenes were still misclassified when compared to general Places365 categories (e.g., "hanbok experience" misclassified as "temple" or "water").
- Focus on Seoul: While Seoul is a rich case study, the findings might not be directly generalizable to all urban tourism contexts without further validation.
- Photo Quality: The "difficult to classify" category acknowledges inherent limitations in photo quality (blurriness, zoom, etc.) that can hinder classification.
- Future Work:
- Multi-label Classification: Developing models that can assign multiple labels to a single photo.
- Hybrid Models: Integrating text data (tags, titles) with photo analysis for a more comprehensive understanding.
- Exploring Other CNN Architectures: Investigating models like DenseNet, ResNet, and Xception for comparison.
- Broader Regional Application: Applying and adapting the methodology to classify tourist photos in other regions beyond Korea.
Summary updated