Analyzing and Predicting GIF Interestingness

Michael Gygli, Mohammad SoleymaniACM Multimedia 2016 · DOI

TL;DR

This paper tackles the challenge of understanding and predicting what makes animated GIFs "interesting" to users. By creating a novel annotated dataset and performing in-depth analysis, they reveal that GIFs of pets and action scenes are generally more engaging than those featuring people, and they develop a predictive model that significantly outperforms existing methods.

What Problem Does This Paper Address?

Animated GIFs have exploded in popularity across the internet, from social media to online journalism. However, despite their widespread use, there's a lack of systematic understanding about what makes a GIF engaging or "interesting" to viewers. Existing approaches to identifying interesting content often rely on social signals like "likes" or "comments," which don't always correlate with genuine visual interest. This paper aims to bridge this gap by exploring the intrinsic visual and affective qualities that drive GIF interestingness and by building a computational model to predict it.

Methods

The authors introduce a novel approach to analyzing GIF interestingness. Their core methodology involves:

  • Dataset Creation: They curated a dataset of 2739 GIFs, sourced from existing datasets like Video2GIF and Tumblr. Crucially, they manually filtered out GIFs with text or pornographic content to focus purely on visual interest.
  • Crowdsourced Annotation: These GIFs were then annotated by human raters on Amazon Mechanical Turk across several scales, including interestingness, aesthetics, arousal, valence (pleasantness), and curiosity. They also included a label for explicit content.
  • Feature Extraction: A diverse set of visual features were extracted from the GIFs, ranging from low-level image statistics (entropy, brightness, contrast) and meta-information (frame rate, duration) to higher-level features like facial expressions, visual sentiments (adjective-noun pairs), and features from pre-trained Convolutional Neural Networks (CNNs) like C3D.
  • Predictive Modeling: They employed a Support Vector Regression (SVR) with an RBF kernel to build a model that predicts GIF interestingness based on these extracted features.

Datasets

  • Video2GIF dataset [9]: Used as a source for sampling GIFs.
  • Tumblr dataset [1]: Another source for GIF sampling, with undersampling applied to balance popularity.
  • Newly Introduced Benchmark: The paper introduces a new annotated dataset of 2739 GIFs, labeled with various affective and interest-related attributes. This dataset is made publicly available.

Key Insights and Ideas

  • Pets and Action Reign Supreme: GIFs featuring pets (especially cats and dogs) and action scenes tend to be rated as more interesting than those featuring people.
  • People are Tricky: While popular memes often involve facial expressions, the general presence of people in GIFs is negatively correlated with interestingness. This suggests a need for more nuanced analysis of facial expressions to capture interesting human-centric GIFs.
  • Affective Dimensions Matter: Curiosity, aesthetics, arousal, and valence all show significant correlations with interestingness, with aesthetic appeal having a particularly strong link.
  • Interest vs. Popularity: While interestingness is correlated with "likes" (a measure of popularity), it's not strongly correlated with re-blogging behavior. This highlights that social signals don't perfectly capture intrinsic visual interest.
  • Visual Sentiment is Key: Surprisingly, visual sentiment features (adjective-noun pairs derived from still images) performed better than spatio-temporal CNN features for predicting GIF interestingness, suggesting that what is depicted is more important than how it moves for capturing interest.

Key Results

  • Predictive Model Performance: Their SVR model, using a combination of visual features, achieved a Spearman's rank correlation of 0.5308 for predicting GIF interestingness.
  • Outperforming Baselines: This performance significantly surpasses existing methods for GIF popularity prediction and even a model trained on image interestingness.
  • Feature Importance: Visual sentiment features and Google API features (like facial expressions) were found to be highly informative.
  • Correlation with Popularity: Interest had a correlation of 0.279 with the number of likes on Tumblr, but only -0.039 with reblogs.

Limitations and Future Work

  • Focus on Visuals: The study deliberately removed GIFs with subtitles to focus on visual interest, meaning the role of dialogue or text-based humor is not explored.
  • Cultural Specificity: Annotations were restricted to workers in the USA to mitigate cultural differences in GIF interpretation, which might limit generalizability.
  • Future Directions: The authors suggest extending the model to predict personalized interest, acknowledging that individual preferences can vary significantly. They also aim to explore the role of facial expressions more deeply for human-centric GIFs.

Summary updated

BibTeX
@inproceedings{gygli2016analyzing,
  author = {Michael Gygli and Mohammad Soleymani},
  title = {{Analyzing and Predicting GIF Interestingness}},
  booktitle = {ACM Multimedia},
  year = {2016},
  doi = {10.1145/2964284.2967195},
}
Figure