A Semantics-Assisted Video Captioning Model Trained With Scheduled Sampling

Haoran Chen, Ke Lin, Alexander Maye, Jianmin Li, Xiaolin Hu

TL;DR

This paper tackles three common pitfalls in video captioning: poor semantic feature extraction, exposure bias during training, and the tendency of models to generate overly short captions. The authors propose a semantics-assisted RNN model trained with a multinomial scheduled sampling strategy and a novel sentence-length-modulated loss function, achieving state-of-the-art performance on the YouTube2Text dataset and highly competitive results on MSR-VTT.

What Problem Does This Paper Address?

Video captioning—automatically generating natural language descriptions for video clips—is a complex multi-modal task requiring both computer vision and natural language processing. While encoder-decoder architectures are standard, existing models suffer from three main limitations:

  1. Weak Semantic Grounding: While adding semantic tags (like objects or actions) helps guide the text generator, existing networks often fail to extract truly meaningful and accurate semantic features from the video.
  2. Exposure Bias: RNNs are typically trained using "Teacher Forcing," where the ground-truth previous word is fed into the model to predict the next word. However, during inference, the model must rely on its own predictions. This discrepancy causes compounding errors during generation.
  3. Overly Short Captions: The standard log-likelihood loss function penalizes longer sentences because each additional word reduces the joint probability of the sequence. Consequently, models learn to "play it safe" by generating excessively short, uninformative captions.

Methods

To address these three issues, the authors introduce a refined encoder-decoder architecture with specific training interventions:

  • High-Quality Semantic Detection Network (SDN): The encoder uses a 2D ConvNet (ResNeXt) for static visual features and a 3D ConvNet (ECO) for spatio-temporal features. Alongside these, they train an SDN as a multi-label classifier to predict the most common nouns, verbs, and adjectives in the video. Crucially, they use mean Average Precision (mAP) to rigorously evaluate and select the best semantic features before feeding them to the decoder.
  • Multinomial Scheduled Sampling: To bridge the gap between training and inference, the authors use scheduled sampling. Instead of always feeding the ground-truth word during training, the model gradually transitions to using its own generated words. Unlike standard greedy approaches (picking the argmax word), they use a multinomial strategy—randomly sampling from the model's output probability distribution. This acts as a regularizer and forces the RNN to explore a wider state space.
  • Sentence-Length-Modulated Loss: To fix the short-caption problem, they introduce a hyperparameter $\beta$ to normalize the loss function by the length of the sentence ($L^\beta$). This reduces the penalty for longer sequences, encouraging the model to output more detailed descriptions.

Datasets

The model was trained and evaluated on two widely used video captioning benchmarks:

  • YouTube2Text (MSVD): A dataset containing 1,970 short YouTube video clips (averaging ~10 seconds), with roughly 40 English descriptions per video.
  • MSR-VTT (Version 1.0): A larger-scale benchmark featuring 10,000 short video segments and 200,000 total descriptions.

Key Insights and Ideas

  • Better semantics equal better captions: The authors proved that the quality of the semantic features (measured via mAP) directly correlates with the final captioning performance. By optimizing the SDN first, the language decoder has a much stronger contextual anchor.
  • Randomness helps RNNs learn better: When applying scheduled sampling, randomly sampling from the predicted word distribution (multinomial) outperformed simply picking the most likely word (argmax). It prevents the model from getting stuck in local minima and reduces overfitting.
  • Length normalization is a delicate balance: Modulating the loss function successfully encouraged longer sentences. However, if the length penalty is removed entirely ($\beta = 1$), the model starts generating redundant, repetitive words (e.g., "a person is mixing a bowl of a bowl"). A balanced hyperparameter ($\beta = 0.7$) yielded the best, most descriptive captions without the repetitive stutter.

Key Results

The proposed model achieved remarkable quantitative results, outperforming many existing architectures:

  • YouTube2Text: Achieved state-of-the-art across all major metrics, scoring 62.4 on BLEU-4 (up from the previous best of 54.5) and 109.7 on CIDEr (up from 95.2).
  • MSR-VTT: Achieved highly competitive, near state-of-the-art results with a BLEU-4 of 45.8 and a CIDEr of 53.2.
  • Qualitative Improvements: The generated captions are noticeably richer. For example, instead of generating a generic "a man is pouring," the length-modulated model successfully generates "a man is pouring eggs into a bowl."

Limitations and Future Work

The authors note a few structural limitations in their current approach. Notably, the model achieves these high scores without using an attention mechanism (which is highly standard in modern sequence-to-sequence tasks) or beam search during decoding (which was abandoned to keep decoding fast).

For future work, the authors suggest that integrating spatio-temporal attention mechanisms with their high-quality visual-semantic features could yield even further improvements in video captioning performance.

Summary updated

Abstract

Given the features of a video, recurrent neural networks can be used to automatically generate a caption for the video. Existing methods for video captioning have at least three limitations. First, semantic information has been widely applied to boost the performance of video captioning models, but existing networks often fail to provide meaningful semantic features. Second, the Teacher Forcing algorithm is often utilized to optimize video captioning models, but during training and inference, different strategies are applied to guide word generation, leading to poor performance. Third, current video captioning models are prone to generate relatively short captions that express video contents inappropriately. Toward resolving these three problems, we suggest three corresponding improvements. First of all, we propose a metric to compare the quality of semantic features, and utilize appropriate features as input for a semantic detection network (SDN) with adequate complexity in order to generate meaningful semantic features for videos. Then, we apply a scheduled sampling strategy that gradually transfers the training phase from a teacher-guided manner toward a more self-teaching manner. Finally, the ordinary logarithm probability loss function is leveraged by sentence length so that the inclination of generating short sentences is alleviated. Our model achieves better results than previous models on the YouTube2Text dataset and is competitive with the previous best model on the MSR-VTT dataset.
BibTeX
@inproceedings{chen2020semantics,
  author = {Haoran Chen and Ke Lin and Alexander Maye and Jianmin Li and Xiaolin Hu},
  title = {{A Semantics-Assisted Video Captioning Model Trained With Scheduled Sampling}},
  booktitle = {Frontiers Robotics AI},
  year = {2020},
  url = {https://openreview.net/forum?id=s1yQYYOwao5},
}

Loading related content...

Figure