Translating Videos to Natural Language Using Deep Recurrent Neural Networks

Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond J. Mooney, Kate Saenko

Abstract

Solving the visual symbol grounding problem has long been a goal of artificial intelligence. The field appears to be advancing closer to this goal with recent breakthroughs in deep learning for natural language grounding in static images. In this paper, we propose to translate videos directly to sentences using a unified deep neural network with both convolutional and recurrent structure. Described video datasets are scarce, and most existing methods have been applied to toy domains with a small vocabulary of possible words. By transferring knowledge from 1.2M+ images with category labels and 100,000+ images with captions, our method is able to create sentence descriptions of open-domain videos with large vocabularies. We compare our approach with recent work using language generation metrics, subject, verb, and object prediction accuracy, and a human evaluation.

Introduction

For humans, describing a short video clip is a trivial task. For machine learning systems, however, it represents a significant "visual symbol grounding" challenge. Historically, video description research has been bifurcated into two limited approaches: narrow-domain models with restricted vocabularies and rigid Subject-Verb-Object (SVO) templates. While templates provide a basic structure, they are inherently brittle and fail to capture the linguistic richness of natural human speech.

A frame from a video clip illustrating the complexity of visual symbol grounding in open-domain scenarios.

The primary obstacle to moving beyond these templates has been the scarcity of labeled video data. Training flexible "pixels-to-sentences" models requires vast amounts of data that were previously unavailable for the video domain. This work addresses this gap by proposing a unified framework that leverages transfer learning from large-scale image datasets to enable open-domain video captioning.

Proposed Methodology

The core of the proposed system is an end-to-end deep learning architecture that combines Convolutional Neural Networks (CNNs) for visual feature extraction and Recurrent Neural Networks (RNNs) for sequence generation.

The unified framework combining CNNs for visual encoding and LSTMs for natural language generation.

Visual Encoding and Mean Pooling

The system samples frames from a video at regular intervals (e.g., one every ten frames). Each frame is processed through a CNN—specifically a variant of AlexNet pre-trained on ImageNet—to extract high-level visual features. To create a fixed-length representation for the entire video, the model performs mean pooling across the feature vectors of all sampled frames. While this ignores temporal ordering, it provides a robust "gist" of the video content, which proves sufficient for short clips depicting single activities.

Sequence Generation via LSTM

The pooled video vector serves as the initial input to a two-layer Long Short-Term Memory (LSTM) network. The LSTM is trained to generate a sentence word-by-word, conditioned on the visual features and the previously generated words. This allows the model to produce varied and natural sentence structures, breaking away from the constraints of SVO templates.

Transfer Learning: The "Secret Sauce"

To overcome the limited size of video datasets, the authors employ a transfer learning strategy. The LSTM is first trained on massive image-captioning datasets such as COCO and Flickr30k. By learning the relationships between objects and language in static images first, the model develops a strong linguistic foundation before being fine-tuned on video-specific data from the YouTube snippets corpus.

Experimental Setup

The model was trained and evaluated using the following datasets:

  • Video Data: The Microsoft Research Video Description Corpus, consisting of approximately 1,970 YouTube snippets with multiple human-written descriptions.
  • Image Data for Transfer Learning: ImageNet (1.2M images) for object recognition, and Flickr30k and COCO (totaling ~110k images) for language grounding.

Results and Discussion

The performance of the model was assessed using standard translation metrics (BLEU and METEOR) alongside human evaluation.

  • Quantitative Performance: The base model trained solely on video data achieved a BLEU score of 31.19. Incorporating transfer learning from the COCO dataset increased this score to 33.29, significantly outperforming prior state-of-the-art template-based models, which typically scored around 13.68.
  • Qualitative and Human Evaluation: Human judges consistently ranked the LSTM-generated sentences as more relevant and natural than those from previous methods. The model successfully described diverse "in-the-wild" content, such as identifying "a soccer player running" or "a man riding a bike."

Limitations and Conclusion

Despite its success, the model has notable limitations. The use of mean pooling results in "temporal blindness," making it difficult for the model to distinguish between actions that differ only in chronological order (e.g., sitting down vs. standing up). Additionally, the generative process occasionally suffers from grammatical repetitions. Future work will focus on more sophisticated temporal modeling to capture complex dynamics and improving language model coherence.

This research demonstrates that knowledge transfer from static images is a powerful tool for video understanding, allowing models to generalize concepts like "cat" or "playing" to moving imagery with minimal video-specific training.

Summary updated

Abstract

Solving the visual symbol grounding problem has long been a goal of artificial intelligence. The field appears to be advancing closer to this goal with recent breakthroughs in deep learning for natural language grounding in static images. In this paper, we propose to translate videos directly to sentences using a unified deep neural network with both convolutional and recurrent structure. Described video datasets are scarce, and most existing methods have been applied to toy domains with a small vocabulary of possible words. By transferring knowledge from 1.2M+ images with category labels and 100,000+ images with captions, our method is able to create sentence descriptions of open-domain videos with large vocabularies. We compare our approach with recent work using language generation metrics, subject, verb, and object prediction accuracy, and a human evaluation.
BibTeX
@inproceedings{venugopalan2015translating,
  author = {Subhashini Venugopalan and Huijuan Xu and Jeff Donahue and Marcus Rohrbach and Raymond J. Mooney and Kate Saenko},
  title = {{Translating Videos to Natural Language Using Deep Recurrent Neural Networks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1173},
}
Figure