SURPRISE! and When to Schedule It

Zhihuan Huang, Shengwei Xu, You Shan, Yuxuan Lu, Yuqing Kong, Tracy Xiao Liu, Grant Schoenebeck

Abstract

This paper investigates the relationship between "surprise"—defined as fluctuations in audience belief regarding match outcomes—and the perceived quality of live-streamed esports. Using a novel Information Flow Elicitation Platform during the 2020 League of Legends World Championship, researchers found that the timing of surprise is critical. While late-game drama significantly boosts viewer ratings, early-game volatility can negatively impact satisfaction. These findings support the "Peak-End Rule" and offer actionable insights for broadcasters to optimize viewer engagement.

Introduction

With billions of hours of live-streamed content consumed annually, content providers like Twitch and tournament organizers are increasingly focused on understanding the drivers of viewer satisfaction. While "excitement" is intuitively linked to quality, quantifying it has historically been difficult, often relying on indirect proxies like audience size. This research addresses this gap by directly eliciting real-time audience beliefs to calculate Information Flow—the measurable change in a viewer’s confidence regarding a game's outcome—and determining its mathematical relationship with subjective quality ratings.

Methodology

The researchers developed an Information Flow Elicitation Platform to transform subjective viewer experiences into quantifiable data. The experimental workflow consisted of three stages:

  1. Pre-game: Subjects indicated their team preference (Blue, Neutral, or Red).
  2. In-game: While watching live matches, participants updated their belief (0% to 100%) of the "blue team's" winning probability. To ensure data integrity, a quadratic scoring rule (Brier score variation) was used; subjects received monetary incentives based on the accuracy of their belief curves relative to the final outcome.
  3. Post-game: Subjects provided a quality rating on a 1–9 scale.
Experimental workflow: subjects provide team preferences, real-time belief updates during the game, and a final quality rating.

Data and Metrics

The study analyzed 76 games from the 2020 League of Legends World Championship (S10). Data was collected from 107 subjects (primarily STEM majors with LoL experience), resulting in 4,566 unique observations.

"Surprise" was mathematically defined as the sum of the absolute changes in the median belief curve, which aggregates the beliefs of all participants. This metric captures the collective shift in confidence as game events unfold.

Quantifying surprise as the sum of absolute changes in the aggregate median belief curve.

Key Results and Insights

The analysis revealed that the impact of surprise is highly dependent on its timing within the match.

The Timing of Information Flow

The most significant finding is that surprise is not a uniform commodity. Surprise occurring in the second half of a game is a strong positive predictor of quality (regression coefficient: 1.743). Conversely, surprise in the first half is negatively correlated with ratings (coefficient: -2.921). This suggests that early-game volatility may be perceived as "messy" or "unskilled" play, whereas late-game shifts are seen as "clutch" performances.

The Peak-End Effect

The data strongly supports the psychological Peak-End Rule, which posits that humans judge experiences based on their most intense point and their conclusion. "End surprise"—defined as the volatility in the final 2.5 minutes—was the single best predictor of a high rating, explaining more variance (Adjusted $R^2 = 0.232$) than the total surprise of the entire match ($R^2 = 0.099$).

"Rubbish Time" and Objectives

The researchers identified a "rubbish time" penalty: periods at the end of a game where the winner is essentially certain (e.g., >70% win probability) correlate with lower ratings. Peak surprise most frequently occurred between the 20 and 30-minute marks, coinciding with contests over major objectives like Baron Nashor, which serve as primary catalysts for belief shifts.

Discussion and Future Work

While this study focuses on surprise (historical movement of the belief curve), future research could explore suspense (the anticipation of future movement). Additionally, the "surprise profiles" identified here may vary across different genres, such as first-person shooters or traditional sports.

For industry practitioners, these insights suggest that broadcasters can maximize perceived quality by "scheduling" information revelation—using camera angles or statistics to emphasize late-game uncertainty. Furthermore, integrating real-time "belief polls" into streaming platforms could serve as a powerful tool for audience engagement, moving beyond simple chat interactions toward a more data-driven viewer experience.

Summary updated

Abstract

Information flow measures, over the duration of a game, the audience's belief of who will win, and thus can reflect the amount of surprise in a game. To quantify the relationship between information flow and audiences' perceived quality, we conduct a case study where subjects watch one of the world's biggest esports events, LOL S10. In addition to eliciting information flow, we also ask subjects to report their rating for each game. We find that the amount of surprise in the end of the game plays a dominant role in predicting the rating. This suggests the importance of incorporating when the surprise occurs, in addition to the amount of surprise, in perceived quality models. For content providers, it implies that everything else being equal, it is better for twists to be more likely to happen toward the end of a show rather than uniformly throughout.
BibTeX
@inproceedings{huang2021surprise,
  author = {Zhihuan Huang and Shengwei Xu and You Shan and Yuxuan Lu and Yuqing Kong and Tracy Xiao Liu and Grant Schoenebeck},
  title = {{SURPRISE! and When to Schedule It}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/36},
}
Figure