Jointly Localizing and Describing Events for Dense Video Captioning

Yehao Li, Ting Yao, Yingwei Pan, Hongyang Chao, Tao Mei

TL;DR

This paper introduces an end-to-end framework for dense video captioning that unifies event localization and sentence generation into a single, globally optimized network. By introducing a novel "descriptiveness regression" module, the model bridges the gap between vision and language, allowing the captioning quality to directly guide and refine temporal event boundaries.

What Problem Does This Paper Address?

Standard video captioning models take a video and generate a single sentence summarizing the entire clip. While this works for short snippets, it fails miserably on longer, real-world videos that contain multiple distinct events.

To solve this, researchers turned to dense video captioning, a two-step challenge:

  1. Identify when interesting events happen (temporal localization).
  2. Describe what happens in each event using natural language.

Historically, this has been treated as a disjointed, two-stage pipeline: a temporal action proposal network generates video segments, and a separate recurrent neural network (RNN) captions them. The authors point out a major flaw in this approach: the localization step operates blindly without knowing whether a proposed video segment actually contains a coherent, "describable" event from a linguistic perspective. This lack of interaction leads to sub-optimal captions and poorly aligned time windows.

Methods

The authors propose a unified architecture that trains both the Temporal Event Proposal (TEP) and Sentence Generation (SG) modules jointly.

Figure 2. An overview of our Dense Video Captioning framework mainly including Temporal Event Proposal (TEP) and Sentence Gener- ation (SG) (better viewed in color). The input video is first encoded into a series of clip-level features via a 3-D CNNs, which are fed into TEP module to produce candidate proposals. The TEP module is employed by integrating the event/background classification to predict event proposal, temporal coordinate regression to refine the temporal boundaries of each proposal, and descriptiveness regression to infer the descriptive complexity of each event, into a single shot detection architecture. After ranking the candidate proposals with regard to both eventness and descriptiveness scores, the top proposals are in turn injected into SG module for sentence generation. The SG module leverages both attributes and reinforcement learning based optimization to enhance captioning.

Key components include:

  • Single-Shot Temporal Localization: Inspired by single-shot object detectors in computer vision (like SSD), the TEP module uses a 1D Convolutional Neural Network over pre-computed 3D CNN video features (C3D or P3D). It simultaneously predicts event/background classifications and regresses temporal boundaries.
  • Descriptiveness Regression: This is the core novelty. The network learns to predict a "descriptiveness score" for each proposal, which estimates how well that segment can be described by the captioning module. During training, the ground-truth target for this score is the actual sentence-level evaluation reward (METEOR score) generated by the captioning module.
  • Descriptiveness-Driven Temporal Attention: Instead of simple mean-pooling over a video segment's frames, the SG module uses the predicted descriptiveness scores of individual clips as an attention mechanism. This focuses the captioning engine on the most informative frames.
  • Attribute-Augmented LSTM with Reinforcement Learning: The sentence generator is an LSTM boosted by high-level semantic attributes (detected categories from the video). It is optimized using Self-Critical Sequence Training (SCST), a reinforcement learning technique that directly maximizes non-differentiable language metrics like METEOR.

Datasets

The framework was trained and evaluated on the ActivityNet Captions dataset, a large-scale benchmark for dense video captioning.

  • Scale: Contains 20,000 videos covering a wide range of complex human activities.
  • Annotations: Features roughly 100,000 total sentences, averaging 3.65 temporally localized sentences per video.
  • Splits: 10,024 videos for training, 4,926 for validation, and 5,044 for testing.

Key Insights and Ideas

  • Language-Aware Localization: Visual cues alone aren't enough to define an event boundary for captioning. An event boundary is inherently tied to semantic narrative; therefore, a proposal mechanism must evaluate if a segment makes sense to describe.
  • The Power of End-to-End Synergy: By making the language evaluation metric (METEOR) backpropagate all the way to the proposal network via descriptiveness regression, the model prevents errors from compounding across a traditional two-stage pipeline.
  • Features Matter: The choice of underlying 3D CNN feature extractor drastically impacts performance. Moving from standard C3D features to Pseudo-3D (P3D) Residual Network features yielded a massive performance leap.

Key Results

The proposed Dense Video Captioning (DVC) model achieved state-of-the-art performance on the ActivityNet Captions benchmark:

  • Official Test Set Record: Achieved a METEOR score of 12.96% using P3D features, capturing the top spot on the official leaderboard at the time of publication.
  • Validation Set Superiority: When using ground-truth proposals, DVC achieved a METEOR score of 10.33%, outperforming the previous state-of-the-art Dense-Captioning Events (DCE) method (8.88%).
  • Human Evaluation: In a blind user study, captions generated by DVC were judged to be better than or equal to human annotations 35.1% of the time, outperforming competing architectures.
Figure 3. Dense video captioning results on ActivityNet Captions validation set. The output temporally localized sentences are gen- erated by 1) Ground Truth, 2) LSTM, and 3) our DVC. We show the results with the highest overlap with ground truth captions.

As seen in Figure 3, DVC provides much more precise descriptions (e.g., identifying "cutting the cat's claws") compared to standard LSTM baselines which often output generic or incorrect phrases (e.g., "brushing the dog's hair").

Limitations and Future Work

While the paper demonstrates a powerful end-to-end framework, a few limitations and areas for future exploration remain:

  • Feature Dependency: The system relies heavily on pre-computed, frozen 3D CNN features. True end-to-end training that backpropagates all the way to raw video pixels remains computationally prohibitive but could yield even better representations.
  • Context Between Events: The current architecture captions each proposal somewhat independently based on local attention. Incorporating a hierarchical language model to better capture the narrative flow and dependencies between consecutive sentences in a video remains an open area for improvement.

Summary updated

Abstract

Automatically describing a video with natural language is regarded as a fundamental challenge in computer vision. The problem nevertheless is not trivial especially when a video contains multiple events to be worthy of mention, which often happens in real videos. A valid question is how to temporally localize and then describe events, which is known as "dense video captioning." In this paper, we present a novel framework for dense video captioning that unifies the localization of temporal event proposals and sentence generation of each proposal, by jointly training them in an end-to-end manner. To combine these two worlds, we integrate a new design, namely descriptiveness regression, into a single shot detection structure to infer the descriptive complexity of each detected proposal via sentence generation. This in turn adjusts the temporal locations of each event proposal. Our model differs from existing dense video captioning methods since we propose a joint and global optimization of detection and captioning, and the framework uniquely capitalizes on an attribute-augmented video captioning architecture. Extensive experiments are conducted on ActivityNet Captions dataset and our framework shows clear improvements when compared to the state-of-the-art techniques. More remarkably, we obtain a new record: METEOR of 12.96% on ActivityNet Captions official test set.
BibTeX
@inproceedings{li2018jointly,
  author = {Yehao Li and Ting Yao and Yingwei Pan and Hongyang Chao and Tao Mei},
  title = {{Jointly Localizing and Describing Events for Dense Video Captioning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2018},
  doi = {10.1109/cvpr.2018.00782},
}
Figure