Beyond Caption To Narrative: Video Captioning With Multiple Sentences

Andrew Shin, Katsunori Ohnishi, Tatsuya Harada

Abstract

Recent advances in image captioning task have led to increasing interests in video captioning task. However, most works on video captioning are focused on generating single input of aggregated features, which hardly deviates from image captioning process and does not fully take advantage of dynamic contents present in videos. We attempt to generate video captions that convey richer contents by temporally segmenting the video with action localization, generating multiple captions from multiple frames, and connecting them with natural language processing techniques, in order to generate a story-like caption. We show that our proposed method can generate captions that are richer in contents and can compete with state-of-the-art method without explicitly using video-level features as input.

Introduction

Standard video captioning models often treat a video as a single, static entity, aggregating features into a single vector to produce a one-sentence summary. While effective for simple clips, this approach fails to capture the dynamic, multi-event nature of real-world video. A single sentence like "A person is cooking" misses the chronological nuance of a sequence where a person chops vegetables, puts them in a pan, and finally stirs the food.

This research addresses two primary hurdles in creating richer descriptions: temporal segmentation and linguistic cohesion. Traditional shot detection based on visual cuts fails in continuous "in-the-wild" footage, and simple concatenation of segment captions often results in repetitive, robotic text. We propose a pipeline that transforms video into a cohesive narrative by identifying action-based boundaries and applying natural language processing (NLP) to link individual descriptions.

Methods

The proposed system utilizes a three-stage pipeline to convert raw video into a narrative structure:

The proposed pipeline for generating narrative video captions.

1. Temporal Segmentation via Action Localization

Instead of relying on visual cuts, the model identifies changes in action. Using a sliding window approach and a classifier trained on the UCF 101 action dataset, the system detects segments where specific actions occur. This allows the model to handle long, unedited shots where the scene remains the same but the activity changes.

2. Caption Generation

For each identified segment, the model extracts the middle frame and processes it through a standard CNN-RNN architecture. Specifically, VGG-16 is used for feature extraction, and an LSTM is employed for language generation to create a candidate sentence for that specific time slice.

3. Narrative Stitching

To ensure the sentences flow like a story rather than a list, two post-processing steps are applied:

  • Backward Coreference Resolution: The system uses Stanford CoreNLP and WordNet to identify when multiple sentences refer to the same subject. Subsequent mentions (e.g., "A man") are replaced with appropriate pronouns (e.g., "He").
  • Connective Word Generation: Using a "Sentence2Vec" approach, the model compares generated sentences to a database of transitions from the Wikicorpus. It then inserts appropriate bridge words like "then" or "afterwards" to establish temporal flow.

Experimental Results

The model was evaluated on three major benchmarks: the Montreal (DVS) dataset, the MPII Movie Description dataset, and the MS Video (Microsoft Video Description) corpus.

Average number of segments generated as a function of the score threshold for Montreal and MPII datasets.

The "narrative" approach consistently outperformed single-frame baselines across BLEU-4, CIDEr, and METEOR metrics. In the Montreal dataset, CIDEr scores improved from 0.070 to 0.089. In the MS Video dataset, BLEU-4 scores rose from 0.043 to 0.063. Notably, when compared against the state-of-the-art winner of the LSMDC challenge, our method produced captions that were approximately four times longer while maintaining comparable accuracy, capturing significantly more information without sacrificing quality.

Discussion and Conclusion

A key insight of this work is that action changes serve as a superior proxy for event boundaries compared to camera cuts. By focusing on coreference and temporal connectives, we move closer to human-like AI storytelling. Interestingly, the model achieves these results using high-quality image features from specific frames rather than complex 3D convolutions or optical flow.

Future work will address current limitations, such as the reliance on a fixed action vocabulary (UCF 101) and the lack of visual identity tracking in coreference resolution. Moving toward unsupervised action localization and integrating face identification would further enhance the robustness and accuracy of the generated narratives.

Summary updated

Abstract

Recent advances in image captioning task have led to increasing interests in video captioning task. However, most works on video captioning are focused on generating single input of aggregated features, which hardly deviates from image captioning process and does not fully take advantage of dynamic contents present in videos. We attempt to generate video captions that convey richer contents by temporally segmenting the video with action localization, generating multiple captions from multiple frames, and connecting them with natural language processing techniques, in order to generate a story-like caption. We show that our proposed method can generate captions that are richer in contents and can compete with state-of-the-art method without explicitly using video-level features as input.
BibTeX
@inproceedings{shin2016beyond,
  author = {Andrew Shin and Katsunori Ohnishi and Tatsuya Harada},
  title = {{Beyond Caption To Narrative: Video Captioning With Multiple Sentences}},
  booktitle = {ICIP},
  year = {2016},
  url = {https://openreview.net/forum?id=d4fuD7tAcC},
}

Loading related content...

Figure