Abstract
Natsuda Laokulrat, Sang Phan, Noriki Nishida, Raphael Shu, Yo Ehara, Naoaki Okazaki, Yusuke Miyao, Hideki Nakayama. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.
2016.
Introduction
Generating natural language descriptions for videos is a complex task that requires capturing temporal dynamics. Unlike static image captioning, video captioning must account for actions that evolve over time. This paper introduces a sequence-to-sequence (Seq2Seq) framework for automatic video captioning that utilizes a temporal attention mechanism. By allowing the model to dynamically focus on specific video frames while generating each word of a description, the authors achieved state-of-the-art results on the Microsoft Video Description (MSVD) dataset at the time of publication.
Methodology
The proposed architecture addresses the limitations of early "mean pooling" approaches, which averaged features from all frames into a single vector, thereby losing critical temporal information. The authors implement a two-layer LSTM encoder-decoder architecture with a dynamic attention mechanism.
- The Encoder (Layer 1): Processes a sequence of frame features extracted via pre-trained CNNs (VGG16 or ResNet) and produces a hidden state for every frame.
- The Decoder (Layer 2): Generates words sequentially. At each time step, it utilizes a context vector which is a weighted sum of the encoder's hidden states.
- Temporal Attention: The model calculates "relevance scores" to determine which frames are most important for the current word being generated. The researchers experimented with four alignment functions: Dot Product, Bilinear, Concatenation, and Sum.
Datasets and Experimental Setup
The model was evaluated on two prominent benchmarks:
- Microsoft Research Video Description (MSVD): 1,970 short YouTube clips with approximately 40 crowd-sourced captions per clip.
- Montreal Video Annotation Dataset (M-VAD): A larger, more challenging dataset of 46,589 movie clips with typically only one professional description per clip.
Results and Discussion
On the MSVD dataset, the system established new performance benchmarks. The choice of visual features and alignment functions proved critical. Moving from VGG16 to ResNet features provided a substantial boost across all metrics, while the bilinear and dot attention functions generally outperformed others on MSVD.
Key quantitative results on MSVD include:
- METEOR Score: 0.326 (using ResNet and Dot attention), surpassing the previous state-of-the-art of 0.298.
- CIDEr Score: 0.750, a significant improvement over previous models.
- BLEU Score: 0.437.
Qualitatively, the attention-based models produced more specific descriptions, such as correctly identifying "pouring sauce" rather than generic "adding sauce." However, the mechanism struggled on the M-VAD dataset, likely due to the high visual diversity and sparse ground-truth captions (one per clip), which made it difficult for attention weights to converge effectively.
Conclusion and Future Work
The integration of temporal attention into a Seq2Seq framework significantly improves video captioning by allowing models to focus on relevant temporal segments. Future work includes incorporating audio features for richer context and integrating text-to-speech systems to assist visually impaired users.
Summary updated