TL;DR
This paper presents a recurrent neural network (RNN) based system for automatically generating textual descriptions of short video clips. By combining frame-level image features, video-level motion features (dense trajectories), and visual content classification outputs, the system achieves improved performance over using any single feature type alone.
What Problem Does This Paper Address?
Generating natural language descriptions for videos is a challenging task in computer vision. While static image captioning has seen progress, extending this to videos requires capturing temporal dynamics and motion. The authors tackle this by building upon existing RNN-based image captioning frameworks and incorporating video-specific features and semantic information from visual classifiers to improve the accuracy and descriptiveness of video captions.
Methods
The proposed system employs a three-stage pipeline:
- Feature Extraction:
- Keyframe Image Features: Activations from the "fc7" layer of pre-trained 16-layer and 19-layer VGG networks, and the 5th Inception module of GoogLeNet, are extracted from a single keyframe per video. These features are augmented with spatial pyramid pooling.
- Video-Specific Features: Dense trajectories are extracted from video clips. These are then quantized into 1000-dimensional histograms, combined with HOG, Motion Boundary (MBHx, MBHy), and HOF descriptors, resulting in 5000-dimensional video features.
- Visual Content Classification: Linear SVMs are trained on the COCO 2014 dataset for 80 object categories. These classifiers are applied to keyframes, and their outputs are combined to form a class membership vector for each keyframe.
- LSTM Caption Generator: A Long Short-Term Memory (LSTM) network is used to learn generative models of sentences. The LSTM can take features from different stages as input:
- Init Features: These are fed into the LSTM at the first time step to initialize its hidden state. This can be keyframe image features or visual content classification outputs.
- Persistent Features: These are fed into the LSTM at every time step, allowing the model to access visual information throughout the generation process. This can be keyframe image features or dense trajectory video features.
- Output Layer: A softmax layer at the output of the LSTM generates a probability distribution over the vocabulary, predicting the next word in the caption. The model is trained to maximize the log probability of the ground truth captions.
Datasets
- COCO 2014: Used for training the visual content classifiers (SVMs).
- LSMDC 2015 (Large Scale Movie Description Challenge): The public test set of this dataset was used for evaluation. This dataset consists of video clips extracted from movies with corresponding natural language descriptions.
Key Insights and Ideas
- Fusion is Key: Combining different types of features (keyframes, video motion, semantic classification) leads to significantly better captioning performance than relying on any single source.
- Persistent Features Matter: Allowing the LSTM to access visual features throughout the caption generation process (persistent features) is crucial for better performance, especially when using video-level features like dense trajectories.
- Video Features Outperform Static Features: Dense trajectory video features, when used as persistent inputs, contribute more to caption quality than solely relying on keyframe-based features.
- Beam Size Impact: Counterintuitively, using a smaller beam size (beam size 1) during inference resulted in better performance. This is attributed to larger beam sizes leading to more generic, less descriptive captions due to penalizing uncertainty.
Key Results
The paper reports results on the LSMDC 2015 public test set using metrics like BLEU-4, CIDER, METEOR, and ROUGE-L.
- Model 9 (cls+traj): This model, which used visual content classification features as init input and dense trajectory video features as persistent input, achieved the best performance among the tested configurations. For example, with beam size 1, it achieved a CIDER score of 0.092 and a METEOR score of 0.058.
- Benefit of Video Features: Model 6 (cls+kf) which used keyframe features and classifier outputs, performed worse than models using dense trajectories (e.g., Model 9).
- Beam Size Effect: Model 9 with beam size 1 achieved a CIDER score of 0.092, while with beam size 5, it dropped to 0.082. This trend was observed across multiple models.
Limitations and Future Work
- Beam Size: The paper notes the counterintuitive finding that smaller beam sizes lead to better results, suggesting that models might be too conservative with larger beam sizes, producing generic captions.
- Sentence Length: The average sentence length decreased significantly with larger beam sizes, indicating a trade-off between diversity and accuracy.
- Future Work: While not explicitly detailed, the success of combining diverse features suggests that further exploration of feature fusion strategies and potentially more sophisticated temporal modeling could lead to even better video captioning systems. The authors also highlight the importance of releasing their source code for reproducibility and further research.
Summary updated