Hierarchy-Dependent Cross-Platform Multi-View Feature Learning for Venue Category Prediction

Shuqiang Jiang, Weiqing Min, Shuhuan Mei

Abstract

This paper introduces HCM-FL, a framework designed to improve venue category prediction from short videos by leveraging cross-platform data and hierarchical taxonomies. By augmenting models trained on Vine videos with filtered images from Foursquare and incorporating a hierarchical prior into the learning process, the authors significantly outperform existing single-platform and "flat" classification methods.

Introduction: The Challenge of Venue Prediction

Predicting a venue category (e.g., "Japanese Restaurant" vs. "Animal Shelter") from visual data is a core task for location-based services. However, researchers face two major hurdles: data scarcity and the inherent noise of social media platforms. While platforms like Foursquare provide a wealth of images, they are often noisy; for instance, an image tagged under a specific venue might be irrelevant to the category's visual essence.

An example of noisy cross-platform data: a whiskey flask that might be incorrectly associated with a non-related venue category.

Furthermore, most models treat venue categories as a flat list, ignoring the natural hierarchy (e.g., Food > Asian Restaurant > Chinese Restaurant). This neglect misses valuable prior knowledge that could help distinguish between similar categories or improve learning for data-sparse classes.

The HCM-FL Framework

The proposed Hierarchy-dependent Cross-platform Multi-view Feature Learning (HCM-FL) framework addresses these issues through a two-stage process:

1. Cross-Platform Transfer Deep Learning (CPTDL)

Instead of simply merging datasets, CPTDL employs a reinforcement strategy to filter noise. An initial VGG-16 network trained on Vine videos is used to "test" Foursquare images. Only images identified with high confidence are retained for fine-tuning. This process is conducted for both objects (ImageNet-pretrained) and scenes (Places-pretrained) to capture multi-view features.

A high-confidence image of a cat, representing the type of relevant data the filtering process aims to retain for categories like 'Animal Shelter'.

2. Multi-View Feature Learning with Hierarchical Venue Structure (MVFL-HVS)

This stage fuses object and scene features while respecting the venue taxonomy. Using a Bayesian model, the parameters of a child category are regularized to remain similar to its parent category. This "knowledge transfer" allows the model to borrow information from related categories, which is particularly beneficial for venue types with limited training samples.

Experimental Results

The authors evaluated HCM-FL using a dataset of 270,145 Vine micro-videos and 190,299 Foursquare images across 188 categories organized into a four-layer tree. Key findings include:

  • Scene vs. Object: Scene-oriented features (Places CNN) proved more discriminative than object-oriented features, as venues like gyms are often defined by layout rather than specific items.
  • Hierarchical Gain: Moving from a flat model to a 3-layer hierarchical model improved the Micro-F1 (Accuracy) from approximately 35% to over 37%.
  • Performance: HCM-FL achieved a Macro-F1 of 18.82%, a significant jump over previous methods like TRUMANN (5.21%) and DARE (16.66%).

Conclusion and Future Work

While HCM-FL sets a new benchmark, challenges remain regarding visually similar categories (e.g., "Italian Restaurant" vs. "Beer Garden") and extreme data imbalance. Future research may incorporate multi-modal data, such as audio and text, and utilize temporal models like LSTMs to better capture the sequential nature of video data.

Summary updated

BibTeX
@article{jiang2019hierarchy,
  author = {Shuqiang Jiang and Weiqing Min and Shuhuan Mei},
  title = {{Hierarchy-Dependent Cross-Platform Multi-View Feature Learning for Venue Category Prediction}},
  journal = {IEEE Trans. Multimedia},
  year = {2019},
  doi = {10.1109/tmm.2018.2876830},
}
Figure