On the Prediction of Flickr Image Popularity by Analyzing Heterogeneous Social Sensory Data

Samah Aloufi, Shiai Zhu, Abdulmotaleb El-SaddikSensors 2017 · OpenReview

TL;DR

This paper tackles the challenge of predicting the popularity of images on social media platforms like Flickr. By analyzing a rich dataset of heterogeneous social sensory data, the authors demonstrate that a combination of visual content, textual information, and social context significantly improves popularity prediction accuracy compared to single-modality approaches.

What Problem Does This Paper Address?

The explosion of social media has blurred the lines between the physical and virtual worlds, with user-generated content acting as a vast source of information about human behavior, preferences, and environments. Predicting the popularity of social media content, specifically images on platforms like Flickr, is a complex problem. Unlike traditional sensor data, social data is characterized by its sheer volume, noise, unstructured nature, and the inherent human semantics and context that require sophisticated analysis. Existing approaches often focus on individual aspects like visual content or social connections, but a comprehensive understanding of how these diverse factors interact to drive image popularity remains an open challenge.

Methods

The authors propose a multi-modal approach to predict image popularity on Flickr. Their methodology involves:

  • Feature Extraction: They extract features from three main categories:
  • Visual Features: This includes low-level features (Color Histogram, LBP, GIST, BoVW), middle-level features (Classemes, Category-Level Attributes, SentiBank), aesthetic features, and high-level deep learning features (FC7, FC8 from a CNN trained on ImageNet).
  • Contextual Features: These capture information about the image owner (number of contacts, uploaded photos, average views) and the image itself (number of groups, group members, tags).
  • Textual Features: Bag-of-Word (BOW) models using TF and TF-IDF are employed to represent image tags, titles, and descriptions.
  • Fusion Techniques: To leverage the complementary nature of these features, they explore several fusion methods:
  • Early Fusion: Concatenating normalized feature vectors.
  • Late Fusion: Averaging normalized output scores from individual models.
  • Borda Count: A rank-based aggregation method.
  • Weighted Fusion: A score-based approach where weights are determined by system performance.
  • Prediction Model: They employ a Ranking Support Vector Machine (Ranking SVM), framing the problem as a pairwise learning-to-rank task. This model learns to predict the relative popularity of images rather than an exact score. They also compare this with Support Vector Regression (SVR) as a baseline.

Datasets

The study utilizes a dataset collected from Flickr groups.

  • Original Dataset: Consists of 1.5 million images uploaded by 90,532 users, collected across 31 topics.
  • Representative Dataset: The Top-1000 most popular images from the original dataset, ranked by views.
  • Random Dataset: A subset of 50,000 images randomly selected from the original dataset, divided into 10 sets of 5000 images each.

Two experimental scenarios were designed:

  • one_per_user: A dataset where one image is randomly selected per user, totaling 89,663 images (30,000 for training, 59,663 for testing). This simulates scenarios like search results where images from different users are compared.
  • personal_collection: A personalized setting where the top 80 users (based on image contributions) are selected, and their images are split into training (60%) and testing (40%) sets. This simulates predicting popularity within a user's own collection.

Key Insights and Ideas

  • Social context is king (but not the only king): The study strongly suggests that factors beyond visual appeal, such as user popularity, group affiliations, and textual descriptions, play a crucial role in an image's visibility and popularity on social media.
  • Groups matter more than direct contacts: Interactions from group members significantly outweigh those from direct contacts, highlighting the importance of community and shared interests in driving engagement.
  • Text is a powerful visibility booster: Descriptive tags and titles are essential for making images discoverable through keyword searches, directly impacting their reach and subsequent popularity.
  • Beauty is subjective and not always popular: While aesthetic features can play a role, especially within a user's own collection, they don't guarantee popularity on a broader social media scale.
  • Multi-modality is the way forward: Combining visual, textual, and social contextual features yields significantly better prediction performance than relying on any single modality. The synergistic effect of these diverse data sources is key.

Key Results

  • Fusion outperforms individual features: Combining features from different modalities consistently led to improved popularity prediction performance across both the "one_per_user" and "personal_collection" settings.
  • Contextual and textual features are highly effective: These features generally outperformed visual features in predicting image popularity, particularly for the "one_per_user" scenario. For instance, contextual features achieved a correlation of 0.55 for predicting interactions in the "one_per_user" setting.
  • Weighted Fusion is a strong performer: Among the fusion techniques, weighted fusion and late fusion showed particularly strong results, especially when combining all modalities.
  • Ranking SVM excels: The Ranking SVM model achieved better performance than SVR, suggesting that learning relative rankings is more robust for this task than predicting exact scores.
  • Best performance achieved with all features: The proposed approach combining visual, contextual, and textual features using Ranking SVM achieved the highest performance, with results like a rank correlation of 0.79 for views and 0.57 for interactions in the "personal_collection" setting.

Limitations and Future Work

  • Limited access to viewer data: The study acknowledges the difficulty in obtaining comprehensive viewer data from Flickr, relying on proxies like comments and favorites.
  • Unidirectional network: Flickr's network is unidirectional, limiting the ability to fully analyze reverse contact information.
  • External factors not considered: The current work focuses on internal factors influencing popularity. Future work could explore the impact of external real-world events.
  • Geographical and cultural influences: The authors suggest investigating how popularity prediction might vary across different geographical regions and cultural backgrounds.
  • Broader applications: Future work could extend this research to predict topic popularity and product sales based on early uploaded images, impacting economic and marketing sectors.

Summary updated

Abstract

The increase in the popularity of social media has shattered the gap between the physical and virtual worlds. The content generated by people or social sensors on social media provides information about users and their living surroundings, which allows us to access a user’s preferences, opinions, and interactions. This provides an opportunity for us to understand human behavior and enhance the services provided for both the real and virtual worlds. In this paper, we will focus on the popularity prediction of social images on Flickr, a popular social photo-sharing site, and promote the research on utilizing social sensory data in the context of assisting people to improve their life on the Web. Social data are different from the data collected from physical sensors; in the fact that they exhibit special characteristics that pose new challenges. In addition to their huge quantity, social data are noisy, unstructured, and heterogeneous. Moreover, they involve human semantics and contextual data that require analysis and interpretation based on human behavior. Accordingly, we address the problem of popularity prediction for an image by exploiting three main factors that are important for making an image popular. In particular, we investigate the impact of the image’s visual content, where the semantic and sentiment information extracted from the image show an impact on its popularity, as well as the textual information associated with the image, which has a fundamental role in boosting the visibility of the image in the keyword search results. Additionally, we explore social context, such as an image owner’s popularity and how it positively influences the image popularity. With a comprehensive study on the effect of the three aspects, we further propose to jointly consider the heterogeneous social sensory data. Experimental results obtained from real-world data demonstrate that the three factors utilized complement each other in obtaining promising results in the prediction of image popularity on social photo-sharing site.
BibTeX
@inproceedings{aloufi2017prediction,
  author = {Samah Aloufi and Shiai Zhu and Abdulmotaleb El-Saddik},
  title = {{On the Prediction of Flickr Image Popularity by Analyzing Heterogeneous Social Sensory Data}},
  booktitle = {Sensors},
  year = {2017},
  url = {https://openreview.net/forum?id=ArTvOYKzuy},
}
Figure