Beyond Views: Measuring and Predicting Engagement in Online Videos

Siqi Wu, Marian-Andrei Rizoiu, Lexing XieICWSM 2018

TL;DR

While "views" tell us how many people clicked a video, they fail to measure how much they actually cared. This paper introduces relative engagement, a duration-calibrated metric that serves as a stable proxy for video quality and can be predicted with high accuracy ($R^2=0.77$) before a video is even uploaded.

What Problem Does This Paper Address?

In the attention economy, "popularity" (view counts) is often conflated with "quality." However, a view only represents the decision to click, not the decision to stay. A video might have millions of views due to a clickbait thumbnail but be abandoned within seconds.

The authors argue that engagement—specifically the average percentage of a video watched—is a much better measure of value for recommender systems and content producers. The challenge is that engagement is heavily biased by video length; naturally, people are more likely to finish a 30-second clip than a 2-hour documentary. To solve this, the researchers sought to create a metric that accounts for duration, remains stable over time, and can be predicted in a "cold-start" scenario (before any user data is collected).

Methods

The researchers proposed a novel framework centered on the Engagement Map.

  1. Metric Definition: They define average watch percentage ($\bar{\mu}_t$) as the total watch time divided by the product of views and video duration.
  2. Relative Engagement ($\bar{\eta}_t$): To remove the bias of video length, they binned 5.3 million videos by duration. A video’s "relative engagement" is its percentile rank within its specific duration bin. For example, a 10-minute video watched 50% of the way through might be in the 90th percentile for its length, giving it a high relative engagement score.
  3. Prediction Pipeline: They used linear regression with L2 regularization to predict engagement using only "cold-start" features (metadata available at upload). Interestingly, they found that predicting the relative engagement first and then mapping it back to the watch percentage yielded much better results than trying to predict the percentage directly.

Datasets

The study utilizes a massive, multi-faceted data collection:

  • TWEETED VIDEOS: A primary dataset of 5.3 million YouTube videos (and 1.2M channels) discovered via Twitter mentions over two months in 2016.
  • Quality Benchmarks: To validate their metrics, they curated three "high-quality" datasets:
  • VEVO VIDEOS: 67k professionally produced music videos.
  • BILLBOARD VIDEOS: 63 top-charting songs.
  • TOP NEWS VIDEOS: 28k videos from reputable news outlets (e.g., CNN, The Young Turks).

Key Insights and Ideas

  • Duration is the Dominant Factor: Video length alone explains 58% of the variance in watch percentage. Without calibrating for duration, engagement metrics are essentially just proxies for "how short is this video?"
  • Engagement is Stable, Popularity is Not: View counts are "bursty" and driven by external promotions or viral spikes. In contrast, the authors found that engagement levels are remarkably consistent over a video's lifetime. If a video is engaging on Day 1, it likely remains engaging on Day 30.
  • The "Quality" Correlation: Relative engagement strongly correlates with expert-defined quality. 70% of Billboard hits had a relative engagement score over 0.9, significantly higher than the average YouTube video.
  • Topic Informativeness: Not all topics are created equal. Topics like "Obama" or "Baseball" are highly informative of engagement levels, whereas broad topics like "Music" or "Books" have high variance and are poor predictors of whether a specific video will hold an audience's attention.

Key Results

  • High Predictability: Using only metadata (title, tags, category) and channel reputation, the model achieved an $R^2$ of 0.77 for predicting average watch percentage.
  • Channel Reputation Matters Most: The strongest predictor of a new video's engagement is the historical performance of the channel that uploaded it ($R^2=0.42$ on its own).
  • Non-linear Mapping: The relationship between duration and watch time is non-linear. By using their "Engagement Map" to linearize this relationship, the researchers improved prediction accuracy significantly compared to standard baseline models.

Limitations and Future Work

  • Aggregate vs. Individual: The study uses aggregate data (average watch time for all users). It does not account for individual user preferences or "who" is watching, which is a critical component for personalized recommendations.
  • Public Data Constraints: Because the data is scraped from public YouTube/Twitter streams, it may suffer from selection bias (only videos that get tweeted are included).
  • Feature Depth: The current model relies on metadata and topics. The authors suggest that future work could incorporate computer vision and audio analysis to "watch" the video and extract deeper features related to production quality and pacing.
  • The "Clickbait" Gap: While this model predicts engagement, it doesn't explicitly model the gap between the click (popularity) and the watch (engagement), which could be used to detect clickbait automatically.

Summary updated

BibTeX
@inproceedings{wu2018beyond,
  author = {Siqi Wu and Marian-Andrei Rizoiu and Lexing Xie},
  title = {{Beyond Views: Measuring and Predicting Engagement in Online Videos}},
  booktitle = {ICWSM},
  year = {2018},
}
Figure