Proceedings of the Workshop on New Frontiers in Summarization
Qing Ping, Chaomei Chen, Enamul Hoque, Giuseppe Carenini, Ottokar Tilk, Tanel Alumäe, Karan Singla, Evgeny A. Stepanov, Ali Orkan Bayer, Giuseppe Riccardi, Michael Völske, Martin Potthast, Shahbaz Syed, Benno Stein 0001, Maxime Peyrard, Teresa Botschen, Iryna Gurevych, Piji Li, † Lidong, Wai Lam, Bing Lidong, Gholipour Ghalandari, Kathleen McCoy, John E. Miller, Antoine J.‐p. Tixier, Giuseppe, Demian, Guo, Han, Hua, Xinyu, Meladianos, Polykarpos, Pasunuru, Ramakanth, Ping Qing, Martin, Rush, Alexander, Karan, Stein, Vazirgiannis, Vlske, Michael, Wang, Lu, R Barzilay, M Elhadad, R Ehren, P Ekman, G Erkan, D Radev, A Haghighi, L Vanderwende, A Hanjalic, L.-q Xu, G Hirst, D St-Onge, A Ikeda, A Kobayashi, H Sakaji, S Masuyama, Kelsey Allen, Raymond Ng, C Conati, B Steichen, D Toker, Siwei Fu, Jian Zhao, Weiwei Cui, Huamin Qu, Shafiq Joty, Mrquez Llus, Jessica Hullman, Nick Diakopoulos, Quentin JonesOpenalex Unknown2017 · DOI · OpenAlex
With the proliferation of video-sharing platforms, the demand for automated video digestion—specifically highlight detection—has grown significantly. This paper, originally presented at the EMNLP 2017 Workshop on New Frontiers in Summarization, proposes an unsupervised framework that leverages crowdsourced "bullet-screen" (time-sync) comments to identify and summarize key video moments. By addressing the inherent challenges of comment lag, semantic sparsity, and subjectivity, the framework utilizes concept-mapped lexical chains and emotion concentration models. Experimental results on large-scale real-world datasets demonstrate that this approach outperforms traditional benchmarks in both highlight detection accuracy and summarization quality.
Introduction
Traditional video highlight detection often relies on low-level audiovisual features, such as sudden increases in volume or rapid motion. However, these features frequently fail to capture the semantic "why" behind a highlight. The emergence of time-sync comments (TSCs)—user remarks that appear at specific timestamps during playback—provides a rich source of human-centric data. Despite their potential, TSCs present three major technical hurdles:
The Lag Effect: Users typically comment after an event occurs, creating a delay between the video shot and the associated text.
Semantic Sparsity: Comments are often brief, consisting of internet slang or emojis (e.g., "23333" for laughter), which complicates standard NLP modeling.
Subjectivity: Defining a "highlight" is inherently personal, making it difficult to establish a universal ground truth.
Methodology
The authors propose a three-stage unsupervised framework designed to transform noisy comment streams into structured highlight reels.
1. Lag-Calibration via Lexical Chains
To synchronize comments with their corresponding video shots, the system employs "lexical chains"—sequences of semantically related words. The framework operates on the "First Comment Rule": it assumes the initial comment in a semantic chain is the most temporally accurate. Subsequent related comments are then "shrunk" back toward that original timestamp, effectively calibrating the lag and clustering the discussion around the actual event.
2. Concept and Emotion Mapping
To overcome the sparsity of internet slang, the researchers developed a custom emotion lexicon tailored for digital culture. Using word2vec embeddings trained on a massive corpus, they mapped diverse terms and slangs into five core emotional categories: happy, anger, sad, fear, and surprise. This allows the system to recognize that a string like "6666" (slang for "awesome") represents a high-intensity emotional response, even if it lacks traditional dictionary definitions.
3. Shot Importance and Summarization
The video is partitioned into shots, each scored for "highlightness" based on three factors:
Comment Intensity: The raw volume of user activity.
Emotion Concentration: The density of specific, strong emotional responses.
Topic Concentration: The degree to which comments focus on a singular concept (e.g., a specific character's entrance).
For the final summary, the system utilizes a modified SumBasic algorithm. Unlike the standard version, this iteration weights emotional words and mapped concepts more heavily, ensuring the generated text summary captures the specific "vibe" of the highlight.
Experimental Setup
The study utilized two primary datasets:
Bilibili Corpus: A collection of 2,108,746 comments from 6,368 long videos used to train word embeddings and the slang-emotion lexicon.
Ground-Truth Dataset: 11 long videos where "gold standard" highlights were defined by fan-made "mix-clips." A shot was considered a true highlight if it appeared in at least two independent fan-made reels.
Key Results
The framework was evaluated against several benchmarks, including simple "Spike-selection" (picking shots based solely on comment volume).
Detection Performance: The integrated model (Spike+L+E+T) achieved an F1-score of 0.3066, a significant improvement over the baseline Spike method (0.2321).
Impact of Calibration: Adding lag-calibration alone improved the precision of the baseline from 0.25 to 0.31, confirming that timing correction is essential for TSC-based analysis.
Summarization Quality: The modified SumBasic approach achieved a ROUGE-1 score of 0.6006, vastly outperforming standard LexRank (0.4325). This indicates the model is highly effective at selecting the specific keywords users employ to describe highlights.
Conclusion and Future Work
This research demonstrates that crowdsourced comments are a powerful proxy for human attention and emotion. By treating internet slang as a primary data point rather than noise, the framework successfully bridges the gap between raw video data and human sentiment.
Future work aims to integrate these text-based insights with multi-modal features (audio and visual cues) and explore real-time processing for live-streaming platforms where data arrives in a continuous, uncalibrated flow.
Summary updated
Abstract
With the prevalence of video sharing, there are increasing demands for automatic video digestion such as highlight detection.Recently, platforms with crowdsourced time-sync video comments have emerged worldwide, providing a good opportunity for highlight detection.However, this task is non-trivial: (1) time-sync comments often lag behind their corresponding shot; (2) time-sync comments are semantically sparse and noisy; (3) to determine which shots are highlights is highly subjective.The present paper aims to tackle these challenges by proposing a framework that (1) uses concept-mapped lexical-chains for lagcalibration; (2) models video highlights based on comment intensity and combination of emotion and concept concentration of each shot; (3) summarize each detected highlight using improved SumBasic with emotion and concept mapping.Experiments on large real-world datasets show that our highlight detection method and summarization method both outperform other benchmarks with considerable margins. 67889:; for number of comments in each highlight summary.Our task is to (1) generate a set of highlight shots () = { % , ' , ( , … , @ }, and (2) highlight summaries Α = { % , ' , ( , … , @ } as close to ground truth as possible.Each highlight summary comprises a subset of all the comments in this shot: 2 = { % , ' , ( , … , @ C }. Number of highlight shots and number of comments in summary 2 are determined by 123142315 and 67889:; respectively. Video Highlight DetectionIn this section, we introduce our framework for highlight detection.Two preliminary tasks are also described, namely construction of global timesync comment word embedding and emotion lexicon. Preliminaries Word-Embedding of Time-Sync Comments 6 ExperimentIn this section, we conduct experiments on large real datasets for highlight detection and summarization.We will describe the data collection process, evaluation metrics, benchmarks and experiment results. DataIn this section, we describe the datasets collected and constructed in our experiments.All datasets and codes will be made publicly available on Github 2 . Crowdsourced Time-sync Comment CorpusTo train the word-embedding described in 4.1.1,we have collected a large corpus of time-sync comment from Bilibli 3 , a content sharing website in China with time-sync comments.The corpus contains 2,108,746 comments, 15,179,132 tokens, 91,745 unique tokens, from 6,368 long videos.Each comment has 7.20 tokens on average.Before training, each comment is first tokenized using Chinse word tokenization package Jieba 4 .Repeating characters in words such as
BibTeX
@inproceedings{ping2017proceedings,
author = {Qing Ping and Chaomei Chen and Enamul Hoque and Giuseppe Carenini and Ottokar Tilk and Tanel Alumäe and Karan Singla and Evgeny A. Stepanov and Ali Orkan Bayer and Giuseppe Riccardi and Michael Völske and Martin Potthast and Shahbaz Syed and Benno Stein and Maxime Peyrard and Teresa Botschen and Iryna Gurevych and Piji Li and † Lidong and Wai Lam and Bing Lidong and Gholipour Ghalandari and Kathleen McCoy and John E. Miller and Qing Ping and Antoine J.‐P. Tixier and Giuseppe and Chen and Gholipour Ghalandari and Demian and Guo and Han and Hoque and Hua and Xinyu and Kathleen Mccoy and Meladianos and Polykarpos and John Miller and Pasunuru and Ramakanth and Peyrard and Ping Qing and Potthast and Martin and Rush and Alexander and Singla and Karan and Stein and Stepanov and Syed and Tilk and Antoine Tixier and Vazirgiannis and Vlske and Michael and Wang and Lu and R Barzilay and M Elhadad and R Ehren and P Ekman and G Erkan and D Radev and A Haghighi and L Vanderwende and A Hanjalic and L.-Q Xu and G Hirst and D St-Onge and A Ikeda and A Kobayashi and H Sakaji and S Masuyama and Kelsey Allen and Giuseppe Carenini and Raymond Ng and C Conati and G Carenini and E Hoque and B Steichen and D Toker and Siwei Fu and Jian Zhao and Weiwei Cui and Huamin Qu and Enamul Hoque and Giuseppe Carenini and Enamul Hoque and Giuseppe Carenini and Enamul Hoque and Giuseppe Carenini and Enamul Hoque and Shafiq Joty and Mrquez Llus and Giuseppe Carenini and Jessica Hullman and Nick Diakopoulos and Quentin Jones},
title = {{Proceedings of the Workshop on New Frontiers in Summarization}},
booktitle = {Openalex Unknown},
year = {2017},
doi = {10.18653/v1/w17-45},
}