Hierarchy-Dependent Cross-Platform Multi-View Feature Learning for Venue Category Prediction

Shuqiang Jiang, Weiqing Min, Shuhuan Mei

TL;DR

This paper introduces HCM-FL, a framework that improves venue category prediction from short videos by leveraging "cross-platform" data and hierarchical taxonomies. By using filtered images from Foursquare to augment models trained on Vine videos and incorporating a hierarchical prior into the learning process, the authors significantly outperform existing single-platform and "flat" classification methods.

What Problem Does This Paper Address?

Predicting a venue category (e.g., "Japanese Restaurant" vs. "Animal Shelter") from visual data is a core task for location-based services and personalization. However, two major challenges persist:

  1. Data Scarcity and Noise: Training deep networks requires massive amounts of data. While platforms like Vine offer videos, they are often short and limited in number. Other platforms like Foursquare have plenty of images, but they are notoriously noisy—an image tagged "Animal Shelter" might just be a photo of a whiskey bottle.
  2. Ignoring Relationships: Most models treat venue categories as a flat list. In reality, venues exist in a natural hierarchy (e.g., Food > Asian Restaurant > Chinese Restaurant). Ignoring this structure means the model misses out on valuable prior knowledge that could help it distinguish between similar categories or learn from data-sparse classes.

Methods

The authors propose the Hierarchy-dependent Cross-platform Multi-view Feature Learning (HCM-FL) framework, which consists of two main stages:

1. Cross-Platform Transfer Deep Learning (CPTDL)

Instead of just dumping Foursquare images into the training set, CPTDL uses a "reinforcement" strategy:

  • Initial Training: A deep network (VGG-16) is first trained on Vine videos.
  • Filtering: This network is used to "test" Foursquare images. Only images that the model identifies with high confidence (top-K probability) are kept, effectively filtering out the noise.
  • Fine-Tuning: The model is then re-trained using these "cleaned" cross-platform images to enhance its feature extraction capabilities.
  • Multi-View: This process is done twice—once for objects (using ImageNet-pretrained weights) and once for scenes (using Places-pretrained weights).

2. Multi-View Feature Learning with Hierarchical Venue Structure (MVFL-HVS)

This stage fuses the object and scene features while respecting the venue taxonomy:

  • Hierarchical Prior: The authors use a Bayesian model where the parameters of a child category (e.g., "Noodle House") are regularized to be similar to its parent category ("Asian Restaurant").
  • Knowledge Transfer: This allows the model to "borrow" information from related categories, which is especially helpful for venue types that have very few training samples.

Datasets

  • Vine Dataset: 270,145 micro-videos (approx. 6 seconds each) across 188 venue categories.
  • Foursquare Dataset: 190,299 images mapped to the same 188 categories.
  • Hierarchy: The categories are organized into a four-layer tree structure (Root -> 10 nodes -> 389 nodes -> 314 nodes -> 52 nodes).

Key Insights and Ideas

  • Scene vs. Object: The results show that scene-oriented features (Places CNN) are generally more discriminative for venue prediction than object-oriented features (ImageNet CNN). This makes intuitive sense: a "Gym" is defined more by its overall layout than by a single specific object.
  • The Power of the Tree: Introducing more hierarchical layers consistently improved performance. The "aha moment" here is that the hierarchy acts as a powerful regularizer; by forcing related categories to share similar feature spaces, the model becomes more robust.
  • Filtering is Essential: Simply adding Foursquare images without filtering (the "Video-Image" baseline) actually hurt performance in some cases due to semantic drift and noise. The CPTDL filtering step was crucial for making cross-platform data useful.

Key Results

The proposed HCM-FL framework outperformed state-of-the-art baselines:

  • Accuracy Boost: HCM-FL achieved a Macro-F1 of 18.82% and a Micro-F1 (Accuracy) of 37.40%.
  • Comparison: This is a significant jump over previous methods like TRUMANN (5.21% Macro-F1) and DARE (16.66% Macro-F1).
  • Hierarchical Gain: Moving from a "flat" model (L=0) to a 3-layer hierarchical model (L=3) improved the Micro-F1 from roughly 35% to over 37%.

Limitations and Future Work

  • Low Inter-class Variation: The model still struggles with visually similar categories, such as "Italian Restaurant" vs. "Beer Garden." In many cases, these venues look identical in short video clips.
  • Data Imbalance: Some categories (like "Beach Bar") had as few as 61 training samples, leading to near-zero accuracy for those specific labels.
  • Future Directions: The authors suggest incorporating multi-modal data, such as audio and text (tags/comments), and using temporal models like LSTMs to better capture the sequential nature of video data rather than relying on mean-pooled keyframes.

Summary updated

Abstract

In this paper, we focus on visual venue category prediction, which can facilitate various applications for location-based service and personalization. Considering the complementarity of different media platforms, it is reasonable to leverage venue-relevant media data from different platforms to boost the prediction performance. Intuitively, recognizing one venue category involves multiple semantic cues, especially objects and scenes and, thus, they should contribute together to venue category prediction. In addition, these venues can be organized in a natural hierarchical structure, which provides prior knowledge to guide venue category estimation. Taking these aspects into account, we propose a Hierarchy-dependent Cross-platform Multi-view Feature Learning (HCM-FL) framework for venue category prediction from videos by leveraging images from other platforms. HCM-FL includes two major components, namely Cross-Platform Transfer Deep Learning (CPTDL) and Multi-View Feature Learning with the Hierarchical Venue Structure (MVFL-HVS). CPTDL is capable of reinforcing the learned deep network from videos using images from other platforms. Specifically, CPTDL first trained a deep network using videos. These images from other platforms are filtered by the learnt network and these selected images are then fed into this learnt network to enhance it. Two kinds of pre-trained networks on the ImageNet and Places dataset are employed. Therefore, we can harness both object-oriented and scene-oriented deep features through these enhanced deep networks. MVFL-HVS is then developed to enable multi-view feature fusion. It is capable of embedding the hierarchical structure ontology to support more discriminative joint feature learning. We conduct the experiment on videos from Vine and images from Foursquare. These experimental results demonstrate the advantage of our proposed framework in jointly utilizing multi-platform data, multi-view deep features, and hierarchical venue structure knowledge.
BibTeX
@article{jiang2019hierarchy,
  author = {Shuqiang Jiang and Weiqing Min and Shuhuan Mei},
  title = {{Hierarchy-Dependent Cross-Platform Multi-View Feature Learning for Venue Category Prediction}},
  journal = {IEEE Trans. Multimedia},
  year = {2019},
  doi = {10.1109/tmm.2018.2876830},
}
Figure