Translating Videos to Natural Language Using Deep Recurrent Neural Networks

Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond J. Mooney, Kate Saenko

This paper, a foundational work in the field of video understanding, presents an end-to-end deep learning architecture that translates video pixels directly into natural language sentences. By leveraging transfer learning from large-scale image captioning datasets, the authors successfully overcome the chronic scarcity of labeled video data, producing descriptions that are more fluid and diverse than previous "fill-in-the-blank" template-based methods.

What Problem Does This Paper Address?

For humans, describing a five-second video clip is trivial. For machines, it is a "visual symbol grounding" nightmare. Historically, researchers tackled this in two ways:

  1. Narrow Domains: Building models that only work for specific scenarios, like a kitchen or a laboratory, with a very limited vocabulary.
  2. SVO Templates: Detecting a Subject, a Verb, and an Object (SVO) separately and then plugging them into a rigid sentence template (e.g., "The [Subject] is [Verb]ing the [Object]").

The problem with templates is that they are brittle and cannot capture the richness of natural language. However, moving to a more flexible "pixels-to-sentences" model requires massive amounts of data—data that simply didn't exist for video at the time. This paper addresses how to build a flexible, open-domain video captioning system despite having very few captioned videos to train on.

Methods

The authors propose a unified framework combining Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

  • Visual Encoding: The system samples frames from a video (one every ten frames) and passes them through a CNN (specifically, a variant of AlexNet pre-trained on ImageNet). To represent the entire video as a single vector, they perform mean pooling across the features extracted from all sampled frames.
  • Sequence Generation: This video vector is fed into a two-layer Long Short-Term Memory (LSTM) network. The LSTM is trained to predict the next word in a sentence given the visual features and the words generated so far.
  • The Secret Sauce (Transfer Learning): Because the video dataset (YouTube snippets) is small, the authors first train their LSTM on massive image-captioning datasets like COCO and Flickr30k. They then "fine-tune" this model on the video data. This allows the model to learn the "logic" of language and object recognition from images before trying to tackle the complexities of video.

Datasets

  • Video: The Microsoft Research Video Description Corpus (YouTube snippets), containing roughly 1,970 clips with multiple human-written descriptions.
  • Image (Transfer Learning): ImageNet (1.2M images for object recognition), and Flickr30k and COCO (totaling ~110k images with captions) to teach the model how to describe static scenes.

Key Insights and Ideas

  • Images Teach Video: The most significant "aha moment" is that a model doesn't need millions of videos to understand video. By learning what a "cat" or "playing" looks like in millions of static images, the model can generalize those concepts to moving pictures with relatively little extra training.
  • Breaking the Template: By using an LSTM to generate text word-by-word, the model can produce varied sentence structures. It isn't forced into a "Subject-Verb-Object" box, allowing for more natural-sounding descriptions.
  • Mean Pooling is Surprisingly Effective: While mean pooling ignores the chronological order of frames, it provides a robust "gist" of the video. For short clips (10–25 seconds) depicting a single activity, the temporal order turns out to be less critical than the overall visual content.

Key Results

The authors evaluated their model using standard translation metrics (BLEU and METEOR) and human judgment.

  • Quantitative Boost: The base model trained only on YouTube data achieved a BLEU score of 31.19. By adding transfer learning from the COCO dataset, the score jumped to 33.29, outperforming previous state-of-the-art template-based models (which scored around 13.68).
  • Human Evaluation: In "blind" tests, humans ranked the LSTM-generated sentences as significantly more relevant to the video content than previous methods.
  • Qualitative Success: The model successfully described "in-the-wild" YouTube videos, such as identifying a "soccer player running" or a "man riding a bike," which were difficult for previous narrow-domain systems.

Limitations and Future Work

  • Temporal Blindness: Because of the mean pooling approach, the model cannot distinguish between a person sitting down and a person standing up (as the frames are the same, just in a different order). The authors acknowledge that more sophisticated temporal modeling is needed for complex actions.
  • Grammatical Hallucinations: Like many early generative models, the LSTM occasionally repeats prepositions or articles (e.g., "a man is playing a a guitar"), indicating a need for better language modeling.
  • Overfitting: In some cases, the transfer learning caused the model to "overfit" to image-style descriptions, sometimes missing specific details unique to the video dynamics.

Summary updated

Abstract

Solving the visual symbol grounding problem has long been a goal of artificial intelligence. The field appears to be advancing closer to this goal with recent breakthroughs in deep learning for natural language grounding in static images. In this paper, we propose to translate videos directly to sentences using a unified deep neural network with both convolutional and recurrent structure. Described video datasets are scarce, and most existing methods have been applied to toy domains with a small vocabulary of possible words. By transferring knowledge from 1.2M+ images with category labels and 100,000+ images with captions, our method is able to create sentence descriptions of open-domain videos with large vocabularies. We compare our approach with recent work using language generation metrics, subject, verb, and object prediction accuracy, and a human evaluation.
BibTeX
@inproceedings{venugopalan2015translating,
  author = {Subhashini Venugopalan and Huijuan Xu and Jeff Donahue and Marcus Rohrbach and Raymond J. Mooney and Kate Saenko},
  title = {{Translating Videos to Natural Language Using Deep Recurrent Neural Networks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1173},
}

Loading related content...

Figure