Ad quality on TV: predicting television audience retention

Yannet Interian, Sundar Dorai-Raj, Igor Naverniouk, P. J. Opalinski, Kaustuv, Dan ZigmondData Mining and Audience Intelligence for Advertising @ KDD 2009 · DOI

TL;DR

This paper introduces a novel method for measuring TV ad quality by analyzing set-top box (STB) tuning data to predict audience retention. By developing a "retention score" that accounts for viewer behavior before and during an ad, the authors demonstrate a more robust measure of ad appeal than traditional survey methods, paving the way for more effective ad targeting and optimization in the TV advertising space.

What Problem Does This Paper Address?

The advertising industry has long sought ways to measure the effectiveness of ads, particularly in the digital realm where metrics like Click-Through Rate (CTR) are standard. However, applying similar concepts to television advertising has been challenging due to the passive nature of TV viewing and the difficulty in isolating the impact of an ad from other factors influencing viewer behavior. Traditional TV ad quality measures, like those from Nielsen, rely on surveys and are limited in scope and granularity. This paper aims to bridge this gap by leveraging large-scale, second-by-second STB tuning data to create a more objective and data-driven measure of TV ad quality – audience retention.

Methods

The core of the paper is the development of a "retention score" (RS). This score is derived from the Initial Audience Retained (IAR), which measures the percentage of viewers who remain tuned to a channel during an ad. To isolate the creative's impact from other factors (like time of day, network, etc.), the authors first calculate an Expected IAR using a logistic regression model. The IAR residual (actual IAR minus Expected IAR) then serves as a measure of the creative's effect.

Two models are proposed for estimating Expected IAR:

  • Model 1: A baseline logistic regression model considering factors like day part, day of the week, network, and creative length.
  • Model 2: An enhanced model that incorporates user behavior, specifically the number of tune-out events in the hour preceding the ad. This model aims to capture a viewer's "mood" or propensity to switch channels.

The retention score (RS) is then calculated as 1 minus the fraction of underperforming airings (those with a negative IAR residual). The authors also validate their retention scores against human evaluations through a survey.

Datasets

The primary dataset used in this research is anonymized, second-by-second tuning data collected from millions of television set-top boxes (STBs) provided by DISH Network L.L.C. This data covers millions of US households and thousands of TV ad airings daily. The authors also conducted a small-scale survey of 78 Google employees to evaluate 20 television ads and compare these human evaluations with their calculated retention scores.

Key Insights and Ideas

  • Viewer behavior matters: STBs exhibiting "active" tuning behavior (frequent channel changes) before an ad are significantly more likely to tune away during that ad, suggesting a viewer's pre-ad state influences their receptiveness.
  • TV ad quality is measurable: Despite the challenges of passive TV viewing, a data-driven retention score can effectively capture an ad's appeal and relevance to viewers.
  • Retention scores correlate with human judgment: The calculated retention scores show a strong correlation with human evaluations of ad "goodness" or "annoyance," validating the metric's effectiveness.
  • Model 2's superiority: Incorporating user behavior (Model 2) significantly improves the stability and predictive power of retention scores compared to a model that only considers contextual factors (Model 1). This highlights the importance of understanding viewer context.
  • Predictive power: Retention scores are strong predictors of future ad performance, enabling advertisers to identify "good" and "bad" ads with high confidence.

Key Results

  • Model Stability: Model 2, which includes user behavior, showed a 27% decrease in the variance of retention scores compared to Model 1 when comparing scores from March to April 2009.
  • Predictive Accuracy: In experiments where "good" and "bad" ads were aired side-by-side, ads with positive retention scores consistently retained more audience than those with negative scores. For ads with a 0.6 difference in retention score, the prediction of which ad would perform better was correct over 90% of the time.
  • Predictive Power Comparison (Table 3): Model 2 was able to label 35% of creatives as either good or bad, compared to 24% for Model 1. The overlap between the models was significant, but Model 2 demonstrated a better ability to distinguish between good and bad ads.
  • Dispersion Number: Model 2 had a dispersion number of 3.4, while Model 1 had 83.7, indicating Model 2 explained 25 times more variability in the data than Model 1.

Limitations and Future Work

  • Limitations: The authors acknowledge that their survey data is from an "admittedly unrepresentative sample" of Google employees. They also note that while their models are robust, there's still "noise" from non-creative factors. The current models do not fully account for audience fatigue or retention at different points within an ad (beginning, middle, end).
  • Future Work: The authors suggest extending their work by incorporating more granular analysis of ad retention throughout its duration, exploring the concept of audience fatigue, and further refining models to better predict viewer behavior. They also hope their methods will inspire more relevant advertising and a more rewarding viewing experience for audiences.

Summary updated

BibTeX
@inproceedings{interian2009ad,
  author = {Yannet Interian and Sundar Dorai-Raj and Igor Naverniouk and P. J. Opalinski and Kaustuv and Dan Zigmond},
  title = {{Ad quality on TV: predicting television audience retention}},
  booktitle = {Data Mining and Audience Intelligence for Advertising @ KDD},
  year = {2009},
  doi = {10.1145/1592748.1592760},
}

Loading related content...

Figure