A Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling

Haoran Chen, Ke Lin, Alexander Maye, Jianming Li, Xiaolin Hu

Abstract

Given the features of a video, recurrent neural networks can be used to automatically generate a caption for the video. Existing methods for video captioning have at least three limitations. First, semantic information has been widely applied to boost the performance of video captioning models, but existing networks often fail to provide meaningful semantic features. Second, the Teacher Forcing algorithm is often utilized to optimize video captioning models, but during training and inference, different strategies are applied to guide word generation, leading to poor performance. Third, current video captioning models are prone to generate relatively short captions that express video contents inappropriately. Toward resolving these three problems, we suggest three corresponding improvements. First of all, we propose a metric to compare the quality of semantic features, and utilize appropriate features as input for a semantic detection network (SDN) with adequate complexity in order to generate meaningful semantic features for videos. Then, we apply a scheduled sampling strategy that gradually transfers the training phase from a teacher-guided manner toward a more self-teaching manner. Finally, the ordinary logarithm probability loss function is leveraged by sentence length so that the inclination of generating short sentences is alleviated. Our model achieves better results than previous models on the YouTube2Text dataset and is competitive with the previous best model on the MSR-VTT dataset.

Introduction

Video captioning—the automatic generation of natural language descriptions for video clips—is a complex multi-modal task at the intersection of computer vision and natural language processing. While encoder-decoder architectures have become standard, existing models frequently struggle with three core issues: weak semantic grounding, exposure bias during training, and a persistent tendency to generate overly short, uninformative captions.

This paper introduces a semantics-assisted RNN model designed to overcome these hurdles. By integrating a high-quality Semantic Detection Network (SDN), employing a multinomial scheduled sampling strategy, and applying a novel sentence-length-modulated loss function, the proposed architecture achieves state-of-the-art performance on the YouTube2Text (MSVD) dataset and highly competitive results on MSR-VTT.

Methods

The proposed model utilizes a refined encoder-decoder architecture that incorporates specific interventions to improve descriptive accuracy and training stability.

High-Quality Semantic Detection Network (SDN)

The encoder extracts visual features using a 2D ConvNet (ResNeXt) for static information and a 3D ConvNet (ECO) for spatio-temporal dynamics. These features are fed into a Semantic Detection Network (SDN), which acts as a multi-label classifier to predict the most relevant nouns, verbs, and adjectives. To ensure the quality of these semantic tags, the authors use mean Average Precision (mAP) to evaluate and select the most effective features before they are passed to the decoder.

Architecture of the semantics-assisted video captioning model, illustrating the integration of 2D/3D ConvNets, the SDN, and the S-LSTM decoder.

Multinomial Scheduled Sampling

Standard RNN training often uses "Teacher Forcing," where the ground-truth word from the previous step is used to predict the next. This creates "exposure bias," as the model never learns to recover from its own errors during inference. To bridge this gap, the authors implement scheduled sampling, where the model gradually transitions from using ground-truth words to using its own predictions. Crucially, they employ a multinomial strategy—sampling from the model's output probability distribution rather than just picking the most likely word—which encourages broader exploration of the state space.

Sentence-Length-Modulated Loss

To counteract the model's bias toward short sentences, the standard log-likelihood loss is normalized by the sentence length $L$ raised to a hyperparameter $eta$. By reducing the penalty for longer sequences, the model is encouraged to generate more detailed and descriptive captions.

Experiments and Results

The model was evaluated on two major benchmarks: YouTube2Text (MSVD) and MSR-VTT. A key finding was that the quality of semantic features (mAP) directly correlates with captioning performance. By optimizing the SDN backbones, the language decoder receives a much stronger contextual anchor.

Comparison of mAP for different semantic feature extractors (ResNeXt, ECO, and their combination) across training epochs.

Key Findings

  • Semantics Matter: Higher mAP in the SDN leads to better BLEU and CIDEr scores. The combination of ECO and ResNeXt features provided the most robust semantic grounding.
  • Sampling Strategy: Multinomial sampling outperformed greedy "argmax" sampling by preventing the model from getting stuck in local minima and reducing overfitting.
  • Balanced Length: A length-modulation hyperparameter of $eta = 0.7$ was found to be the "sweet spot," encouraging descriptive detail without introducing the repetitive redundancy seen at higher $eta$ values.

On the YouTube2Text dataset, the model achieved a BLEU-4 of 62.4 and a CIDEr of 109.7, setting a new state-of-the-art. On MSR-VTT, it remained highly competitive with a BLEU-4 of 45.8.

Conclusion

By addressing the fundamental weaknesses of semantic extraction, training bias, and length penalties, this research demonstrates that relatively simple architectural and training interventions can significantly boost video captioning quality. Future work may involve integrating spatio-temporal attention mechanisms to further refine the alignment between visual features and generated text.

Summary updated

Abstract

Given the features of a video, recurrent neural networks can be used to automatically generate a caption for the video. Existing methods for video captioning have at least three limitations. First, semantic information has been widely applied to boost the performance of video captioning models, but existing networks often fail to provide meaningful semantic features. Second, the Teacher Forcing algorithm is often utilized to optimize video captioning models, but during training and inference, different strategies are applied to guide word generation, leading to poor performance. Third, current video captioning models are prone to generate relatively short captions that express video contents inappropriately. Toward resolving these three problems, we suggest three corresponding improvements. First of all, we propose a metric to compare the quality of semantic features, and utilize appropriate features as input for a semantic detection network (SDN) with adequate complexity in order to generate meaningful semantic features for videos. Then, we apply a scheduled sampling strategy that gradually transfers the training phase from a teacher-guided manner toward a more self-teaching manner. Finally, the ordinary logarithm probability loss function is leveraged by sentence length so that the inclination of generating short sentences is alleviated. Our model achieves better results than previous models on the YouTube2Text dataset and is competitive with the previous best model on the MSR-VTT dataset.
BibTeX
@inproceedings{chen2020semantics,
  author = {Haoran Chen and Ke Lin and Alexander Maye and Jianming Li and Xiaolin Hu},
  title = {{A Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling}},
  booktitle = {Frontiers Robotics AI},
  year = {2020},
  url = {https://openreview.net/forum?id=s1yQYYOwao5},
}
Figure