Introduction
The explosion of live streaming platforms like Twitch and YouTube Live has created a massive archive of video content. However, for most viewers, the value lies in "highlights"—brief, high-intensity moments such as game-winning plays or dramatic team fights. Traditional methods for highlight extraction typically rely on computationally expensive deep learning models or costly manual tagging. This paper introduces LIGHTOR, a system that leverages "implicit crowdsourcing" to identify highlights by analyzing the natural behaviors of the audience: their chat messages and their video playback interactions.
The Challenge of Live Stream Highlights
Existing solutions for highlight extraction generally fall into two categories:
- Algorithmic/Deep Learning Models: While powerful, models like CNNs or LSTMs require vast amounts of labeled training data and are often game-specific. A model optimized for Dota 2 rarely generalizes to League of Legends without significant retraining.
- Explicit Crowdsourcing: This involves paying humans to manually tag videos. While accurate, it is prohibitively expensive and cannot scale to the millions of hours of content generated daily.
LIGHTOR addresses these issues by treating viewers not just as consumers, but as implicit data annotators. By monitoring how they chat and how they navigate the video player, the system can pinpoint exciting moments with high precision and minimal computational overhead.
The LIGHTOR System Architecture
LIGHTOR operates through a two-stage workflow designed to first predict and then refine highlight locations.
1. Highlight Initializer (The Chat Phase)
The first stage analyzes live chat logs to identify potential highlights. Rather than processing pixels, LIGHTOR focuses on three behavioral features:
- Message Volume: Sudden spikes in chat frequency often correlate with significant events.
- Message Length: During intense moments, viewers tend to send shorter, punchier messages (e.g., emojis or "OMG").
- Message Similarity: Highlights often produce a "consensus" in the chat, where many users send identical or highly similar emotes.
To account for the natural "reaction lag" between an in-game event and the audience's response, the system includes an Adjustment Stage. By learning typical delays (often around 20 seconds), LIGHTOR places initial markers—"red dots"—on the video's progress bar.
2. Highlight Extractor (The Interaction Phase)
The second stage refines these initial guesses using viewer interaction data. When a "red dot" is placed on the progress bar, it acts as a nudge for viewers. LIGHTOR monitors how users respond to these hints:
- Validation: If a user clicks a dot and continues watching, the highlight is likely valid. If they immediately skip away, it is flagged as a false positive.
- Boundary Refinement: If users consistently "seek backward" after clicking a dot, the system recognizes that the marker was placed too late and iteratively adjusts the start and end timestamps of the clip.
Evaluation and Results
The researchers evaluated LIGHTOR using datasets from Dota 2 and League of Legends, involving nearly 500 participants from Amazon Mechanical Turk.
As shown in the results, LIGHTOR achieves a Precision@K of 70% to 90% for top-tier highlights. More impressively, the system is exceptionally efficient: it requires 123x fewer training examples and 100,000x less training time than state-of-the-art Joint-LSTM models. Because it relies on human behavior rather than visual features, LIGHTOR also exhibits superior generalization; a model trained on one game performs effectively on others without modification.
Conclusion and Future Work
LIGHTOR demonstrates that implicit crowdsourcing is a viable, scalable alternative to traditional video analysis. By turning the audience's natural reactions into data, it creates a self-refining loop for highlight extraction. Future research will focus on improving the delay model and implementing semantic filtering to distinguish between gameplay highlights and non-gameplay "meta" moments (e.g., funny commentary during breaks).
Summary updated