Video Captioning with Listwise Supervision

Yuan Liu, Xue Li, Zhongchao ShiAAAI 2017 · DOI

Introduction

Video captioning—the automated generation of natural language descriptions for video content—is a complex challenge requiring the integration of spatial and temporal visual information. Traditional "sequence-to-sequence" models typically learn from isolated video-sentence pairs. This "point-wise" approach fails to capture the broader semantic relationships between different descriptions, often leading to a lack of precision in word choice. For example, a model might struggle to determine if "a man is riding a motorcycle" is a more appropriate description than "a man is riding a bike" without understanding their relative relevance to the visual features.

Methodology: LSTM-LS

To address these limitations, this paper introduces Long Short-Term Memory with Listwise Supervision (LSTM-LS). This framework shifts the training paradigm from absolute correctness to relative appropriateness by employing a ranked list of sentences for supervision.

A sample video sequence of a person riding a motorcycle, illustrating the type of content the model aims to describe and rank.

Feature Extraction

The model utilizes a dual-stream encoder to capture comprehensive visual data:

  • Spatial Features: Extracted using 2D Convolutional Neural Networks (VGG).
  • Temporal Features: Captured via 3D CNNs (C3D) to model motion. These features are mean-pooled into a fixed-length vector that serves as the input to the LSTM decoder.

Listwise Supervision

The core innovation of the framework is the supervision strategy. For every ground-truth sentence associated with a video, the authors perform a nearest-neighbor search to identify the top $K$ semantically similar sentences from the entire training set. This creates a "ground-truth ranking list." A custom Listwise Loss Layer then trains the model to minimize the difference between its predicted log-probabilities and this ground-truth ranking, using a triplet-based representation. This ensures the model learns the relative strength of various descriptions rather than treating them as binary correct/incorrect labels.

Experimental Evaluation

The LSTM-LS model was evaluated on two benchmark datasets: the MSVD (Microsoft Research Video Description Corpus) and the M-VAD (Montreal Video Annotation Dataset).

Performance Metrics

  • MSVD Dataset: The model achieved state-of-the-art results with 51.1% BLEU@4 and 32.6% METEOR, significantly outperforming previous models such as S2VT and LSTM-E.
  • M-VAD Dataset: The model reached a METEOR score of 6.9%, the highest reported at the time of publication.

Key Insights

Ablation studies confirmed that combining 2D and 3D features is essential for accurate captioning, as both "what is there" and "how it moves" are critical. Furthermore, the researchers found that using $K=4$ sentences in the ranking list provided the optimal balance of semantic context and signal-to-noise ratio.

Conclusion

By teaching models to rank descriptions rather than just classify them, LSTM-LS achieves a more nuanced, human-like understanding of visual properties. While the current model uses mean pooling, future iterations could benefit from more sophisticated temporal modeling, such as RNN-based encoders, and the integration of large-scale image captioning data to further boost performance.

Summary updated

BibTeX
@inproceedings{liu2017video,
  author = {Yuan Liu and Xue Li and Zhongchao Shi},
  title = {{Video Captioning with Listwise Supervision}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.1609/aaai.v31i1.11239},
}
Figure