TL;DR
This paper introduces HCM-FL, a framework that improves venue category prediction from short videos by leveraging "cross-platform" data and hierarchical taxonomies. By using filtered images from Foursquare to augment models trained on Vine videos and incorporating a hierarchical prior into the learning process, the authors significantly outperform existing single-platform and "flat" classification methods.
What Problem Does This Paper Address?
Predicting a venue category (e.g., "Japanese Restaurant" vs. "Animal Shelter") from visual data is a core task for location-based services and personalization. However, two major challenges persist:
- Data Scarcity and Noise: Training deep networks requires massive amounts of data. While platforms like Vine offer videos, they are often short and limited in number. Other platforms like Foursquare have plenty of images, but they are notoriously noisy—an image tagged "Animal Shelter" might just be a photo of a whiskey bottle.
- Ignoring Relationships: Most models treat venue categories as a flat list. In reality, venues exist in a natural hierarchy (e.g., Food > Asian Restaurant > Chinese Restaurant). Ignoring this structure means the model misses out on valuable prior knowledge that could help it distinguish between similar categories or learn from data-sparse classes.
Methods
The authors propose the Hierarchy-dependent Cross-platform Multi-view Feature Learning (HCM-FL) framework, which consists of two main stages:
1. Cross-Platform Transfer Deep Learning (CPTDL)
Instead of just dumping Foursquare images into the training set, CPTDL uses a "reinforcement" strategy:
- Initial Training: A deep network (VGG-16) is first trained on Vine videos.
- Filtering: This network is used to "test" Foursquare images. Only images that the model identifies with high confidence (top-K probability) are kept, effectively filtering out the noise.
- Fine-Tuning: The model is then re-trained using these "cleaned" cross-platform images to enhance its feature extraction capabilities.
- Multi-View: This process is done twice—once for objects (using ImageNet-pretrained weights) and once for scenes (using Places-pretrained weights).
2. Multi-View Feature Learning with Hierarchical Venue Structure (MVFL-HVS)
This stage fuses the object and scene features while respecting the venue taxonomy:
- Hierarchical Prior: The authors use a Bayesian model where the parameters of a child category (e.g., "Noodle House") are regularized to be similar to its parent category ("Asian Restaurant").
- Knowledge Transfer: This allows the model to "borrow" information from related categories, which is especially helpful for venue types that have very few training samples.
Datasets
- Vine Dataset: 270,145 micro-videos (approx. 6 seconds each) across 188 venue categories.
- Foursquare Dataset: 190,299 images mapped to the same 188 categories.
- Hierarchy: The categories are organized into a four-layer tree structure (Root -> 10 nodes -> 389 nodes -> 314 nodes -> 52 nodes).
Key Insights and Ideas
- Scene vs. Object: The results show that scene-oriented features (Places CNN) are generally more discriminative for venue prediction than object-oriented features (ImageNet CNN). This makes intuitive sense: a "Gym" is defined more by its overall layout than by a single specific object.
- The Power of the Tree: Introducing more hierarchical layers consistently improved performance. The "aha moment" here is that the hierarchy acts as a powerful regularizer; by forcing related categories to share similar feature spaces, the model becomes more robust.
- Filtering is Essential: Simply adding Foursquare images without filtering (the "Video-Image" baseline) actually hurt performance in some cases due to semantic drift and noise. The CPTDL filtering step was crucial for making cross-platform data useful.
Key Results
The proposed HCM-FL framework outperformed state-of-the-art baselines:
- Accuracy Boost: HCM-FL achieved a Macro-F1 of 18.82% and a Micro-F1 (Accuracy) of 37.40%.
- Comparison: This is a significant jump over previous methods like TRUMANN (5.21% Macro-F1) and DARE (16.66% Macro-F1).
- Hierarchical Gain: Moving from a "flat" model (L=0) to a 3-layer hierarchical model (L=3) improved the Micro-F1 from roughly 35% to over 37%.
Limitations and Future Work
- Low Inter-class Variation: The model still struggles with visually similar categories, such as "Italian Restaurant" vs. "Beer Garden." In many cases, these venues look identical in short video clips.
- Data Imbalance: Some categories (like "Beach Bar") had as few as 61 training samples, leading to near-zero accuracy for those specific labels.
- Future Directions: The authors suggest incorporating multi-modal data, such as audio and text (tags/comments), and using temporal models like LSTMs to better capture the sequential nature of video data rather than relying on mean-pooled keyframes.
Summary updated