Abstract
Given the features of a video, recurrent neural networks can be used to automatically generate a caption for the video. Existing methods for video captioning have at least three limitations. First, semantic information has been widely applied to boost the performance of video captioning models, but existing networks often fail to provide meaningful semantic features. Second, the Teacher Forcing algorithm is often utilized to optimize video captioning models, but during training and inference, different strategies are applied to guide word generation, leading to poor performance. Third, current video captioning models are prone to generate relatively short captions that express video contents inappropriately. Toward resolving these three problems, we suggest three corresponding improvements. First of all, we propose a metric to compare the quality of semantic features, and utilize appropriate features as input for a semantic detection network (SDN) with adequate complexity in order to generate meaningful semantic features for videos. Then, we apply a scheduled sampling strategy that gradually transfers the training phase from a teacher-guided manner toward a more self-teaching manner. Finally, the ordinary logarithm probability loss function is leveraged by sentence length so that the inclination of generating short sentences is alleviated. Our model achieves better results than previous models on the YouTube2Text dataset and is competitive with the previous best model on the MSR-VTT dataset.
Introduction
Video captioning—the automatic generation of natural language descriptions for video clips—is a complex multi-modal task at the intersection of computer vision and natural language processing. While encoder-decoder architectures have become standard, existing models frequently struggle with three core issues: weak semantic grounding, exposure bias during training, and a persistent tendency to generate overly short, uninformative captions.
This paper introduces a semantics-assisted RNN model designed to overcome these hurdles. By integrating a high-quality Semantic Detection Network (SDN), employing a multinomial scheduled sampling strategy, and applying a novel sentence-length-modulated loss function, the proposed architecture achieves state-of-the-art performance on the YouTube2Text (MSVD) dataset and highly competitive results on MSR-VTT.
Methods
The proposed model utilizes a refined encoder-decoder architecture that incorporates specific interventions to improve descriptive accuracy and training stability.
High-Quality Semantic Detection Network (SDN)
The encoder extracts visual features using a 2D ConvNet (ResNeXt) for static information and a 3D ConvNet (ECO) for spatio-temporal dynamics. These features are fed into a Semantic Detection Network (SDN), which acts as a multi-label classifier to predict the most relevant nouns, verbs, and adjectives. To ensure the quality of these semantic tags, the authors use mean Average Precision (mAP) to evaluate and select the most effective features before they are passed to the decoder.
Multinomial Scheduled Sampling
Standard RNN training often uses "Teacher Forcing," where the ground-truth word from the previous step is used to predict the next. This creates "exposure bias," as the model never learns to recover from its own errors during inference. To bridge this gap, the authors implement scheduled sampling, where the model gradually transitions from using ground-truth words to using its own predictions. Crucially, they employ a multinomial strategy—sampling from the model's output probability distribution rather than just picking the most likely word—which encourages broader exploration of the state space.
Sentence-Length-Modulated Loss
To counteract the model's bias toward short sentences, the standard log-likelihood loss is normalized by the sentence length $L$ raised to a hyperparameter $eta$. By reducing the penalty for longer sequences, the model is encouraged to generate more detailed and descriptive captions.
Experiments and Results
The model was evaluated on two major benchmarks: YouTube2Text (MSVD) and MSR-VTT. A key finding was that the quality of semantic features (mAP) directly correlates with captioning performance. By optimizing the SDN backbones, the language decoder receives a much stronger contextual anchor.
Key Findings
- Semantics Matter: Higher mAP in the SDN leads to better BLEU and CIDEr scores. The combination of ECO and ResNeXt features provided the most robust semantic grounding.
- Sampling Strategy: Multinomial sampling outperformed greedy "argmax" sampling by preventing the model from getting stuck in local minima and reducing overfitting.
- Balanced Length: A length-modulation hyperparameter of $eta = 0.7$ was found to be the "sweet spot," encouraging descriptive detail without introducing the repetitive redundancy seen at higher $eta$ values.
On the YouTube2Text dataset, the model achieved a BLEU-4 of 62.4 and a CIDEr of 109.7, setting a new state-of-the-art. On MSR-VTT, it remained highly competitive with a BLEU-4 of 45.8.
Conclusion
By addressing the fundamental weaknesses of semantic extraction, training bias, and length penalties, this research demonstrates that relatively simple architectural and training interventions can significantly boost video captioning quality. Future work may involve integrating spatio-temporal attention mechanisms to further refine the alignment between visual features and generated text.
Summary updated