Abstract
Video paragraph captioning aims to generate a coherent story describing a long video. Traditional methods rely on a two-stage pipeline of event detection followed by captioning, which often suffers from a performance bottleneck due to inaccurate event boundaries. This paper introduces VPCSum, a framework that reframes the task as an extractive text summarization problem. By generating a dense pool of candidate captions and selecting the most representative ones using a BERT-based summarizer with visual weighting, VPCSum achieves state-of-the-art performance on ActivityNet and YouCookII benchmarks without requiring ground-truth event boundaries.
The Bottleneck of Event Detection
Describing a long video with multiple events—such as a gymnastics routine or a cooking tutorial—requires a "paragraph caption." The conventional approach follows a rigid two-step pipeline: first, detect specific event segments (Event Detection), and then write a sentence for each (Event Captioning).
This creates a major bottleneck: if the event detection is slightly off, the resulting captions are poor. Because event detection is notoriously difficult, many researchers rely on human-annotated ground-truth segments during testing to achieve reasonable results. The authors of this paper argue that we should instead treat the problem as one of information density. By generating hundreds of possible captions for every part of the video, the task becomes selecting the best subset to form a concise, non-redundant paragraph.
The VPCSum Framework
The proposed VPCSum framework consists of three main modules designed to bypass the detection bottleneck:
- Proposal Extraction: Instead of picking a few "perfect" segments, a Boundary-Matching Network (BMN) extracts the top 100 dense event proposals for every video.
- Proposal Captioning: A candidate sentence is generated for every single one of those 100 proposals using standard captioning models.
- Caption Summarization: This module treats the 100 sentences as a document to be summarized. A BERT-based model scores each sentence based on its likelihood of belonging in the final paragraph.
To optimize the summarization, the authors employ Soft-Labeling, training the model using the ROUGE score (a text overlap metric) between the candidate and the human-written "gold" paragraph. Furthermore, they incorporate Visual Weighting from a pointer network to ensure chosen segments are visually significant, and Trigram Blocking to eliminate linguistic redundancy.
Key Results and Insights
VPCSum was evaluated on the ActivityNet Captions and YouCookII benchmarks. On ActivityNet, VPCSum achieved a BLEU@4 score of 10.89 and a CIDEr score of 24.33, outperforming baselines like MART and COOT. Notably, this extractive method outperformed several models that had access to human-defined ground-truth segments, suggesting that summarization is a more robust approach than explicit detection.
One of the most significant improvements was in the R@4 (repetition) metric. VPCSum scored 1.54, compared to the next best baseline at 6.09, indicating that the generated paragraphs are much less repetitive. Human evaluations confirmed these findings, with judges consistently rating VPCSum higher for both relevance and diversity.
Limitations and Future Work
While the model excelled on ActivityNet, it faced challenges on YouCookII, which features a higher average number of segments per video (7.7 vs. 3.65). This increased density makes the selection task harder for an extractive summarizer. Future work may involve moving toward abstractive summarization, allowing the model to rephrase or combine ideas into entirely new sentences for even more fluid video descriptions.
Summary updated