A Multimodal Approach to Predict Social Media Popularity

Mayank Meghawat, Satyendra Yadav, Debanjan Mahata, Yifang Yin, Rajiv Ratn Shah, Roger Zimmermann

Predicting which photos will go viral and which will languish in obscurity is a "holy grail" for advertisers and content creators alike. While many researchers have tried to solve this using either image analysis or text metadata, a new paper from researchers at the Nirma Institute of Technology, IIIT-Delhi, and the National University of Singapore argues that you need the "whole picture"—literally and figuratively—to get it right.

Overview

The authors propose a multimodal framework that predicts the popularity of Flickr photos by combining visual features, textual metadata (titles, tags, descriptions), and social context. By augmenting the existing SMP-T1 dataset with actual image content and sentiment scores, they achieved state-of-the-art performance using only half the training data of previous models.

The Multimodal Framework

Popularity prediction is notoriously difficult because "popularity" isn't just a property of the image itself; it’s a result of the interaction between the content, the context (what the user says about it), and the social environment (who the user is).

Existing benchmarks, like the ACM Multimedia 2017 Grand Challenge (SMP-T1) dataset, provided metadata like view counts and group memberships but lacked the actual raw images and full textual descriptions for many entries. The researchers developed a three-pronged approach to capture this complexity:

  1. Contextual and Social (CON-SOC) Analysis: They used 14 features, including user average views and group counts, processed via 1D-Convolutional Neural Networks (Conv1D).
  2. Visual Feature Extraction: Using Transfer Learning, they employed a pre-trained InceptionResNetV2 model to extract "popularity scores" directly from image pixels.
  3. Textual Enrichment: They introduced a dictionary-based scoring technique for tags and titles and used sentiment analysis to turn subjective text into numerical features.
Overall Multimodal Framework

Sentiment and Textual Analysis

By adding sentiment analysis of descriptions, the authors were able to capture the "vibe" of a post, which correlates with how likely people are to engage with it. Interestingly, "tag-score" and "title-score" contributed approximately 10% and 18% respectively to the model's predictive accuracy, suggesting that how a user describes their content is nearly as important as the content itself.

Sentiment analysis process for Description Score

Results and Performance

The study utilizes the multimodal-SMP-dataset, a new benchmark created by the authors. Because nearly half of the original image links in SMP-T1 were broken, the authors crawled and recovered 200,000 images, supplementing them with raw image content and full textual metadata.

Performance was measured using Spearman’s Rho (a correlation coefficient where 1.0 is a perfect match):

  • Images only: Performed poorly (Spearman Rho: 0.27), proving pixels alone cannot predict popularity.
  • Metadata only: The CNN on CON-SOC info achieved a Rho of 0.73.
  • Multimodal Fusion: Combining images and metadata pushed the performance to 0.75.

Remarkably, the multimodal approach reached this level of performance using only 200,000 training samples, compared to other state-of-the-art systems that required the full 432,000 samples. This efficiency demonstrates that integrating diverse data types provides a more robust signal for popularity than sheer volume of single-modality data.

Future Work

For future work, the authors suggest expanding modalities to include video data, exploring temporal dynamics (such as the time of day a post is made), and deeper modeling of user "authority" or celebrity status within specific social sub-groups.

Summary updated

BibTeX
@inproceedings{meghawat2018multimodal,
  author = {Mayank Meghawat and Satyendra Yadav and Debanjan Mahata and Yifang Yin and Rajiv Ratn Shah and Roger Zimmermann},
  title = {{A Multimodal Approach to Predict Social Media Popularity}},
  booktitle = {MIPR},
  year = {2018},
  url = {https://openreview.net/forum?id=nykhWETmfqq},
}
Figure