Jointly Modeling Embedding and Translation to Bridge Video and Language

Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, Yong Rui

TL;DR

The LSTM-E framework improves video captioning by jointly optimizing for two goals: generating fluent word sequences (coherence) and ensuring the entire sentence matches the video's global meaning (relevance). By bridging the gap between local word transitions and global visual-semantic context, it achieves state-of-the-art performance in translating complex video dynamics into accurate natural language.

What Problem Does This Paper Address?

Generating natural language descriptions for videos is a "holy grail" of multimedia research. While Recurrent Neural Networks (RNNs) have become the standard for this task, they often suffer from a lack of semantic grounding.

Standard models focus on coherence: predicting the next word based on previous words and visual cues. However, they frequently fail at relevance: the generated sentence might sound like perfect English but describe the wrong action or subject (e.g., saying "a man is riding a horse" when the video clearly shows a woman). This happens because the model doesn't holistically evaluate if the entire sentence makes sense given the entire video.

Methods

The authors propose LSTM-E (Long Short-Term Memory with visual-semantic Embedding), a unified framework that combines sequence learning with a joint embedding space.

  1. Visual Representation: The model uses 2-D CNNs (like VGG) to capture spatial information from frames and 3-D CNNs (C3D) to capture temporal dynamics (motion) from video clips. These features are combined via mean pooling into a single video vector.
  2. The Dual-Loss Objective: This is the core innovation. The model is trained to minimize two types of loss simultaneously:
  3. Coherence Loss: A standard LSTM-based RNN learns the probability of word sequences, ensuring the output is grammatically correct.
  4. Relevance Loss: Both the video vector and the sentence vector (represented by TF weights of words) are projected into a shared low-dimensional embedding space. The model is penalized if the distance between the video and the sentence in this space is too large.
  5. Joint Learning: A tradeoff parameter ($\lambda$) balances these two losses, allowing the model to learn a representation that is both linguistically fluent and semantically grounded.
Figure 2. An overview of our LSTM-E framework with a language generating LSTM and a visual-semantic embedding model (better viewed in color). The video representation is produced by mean pooling over the visual features of frames/clips, extracted by a 2-D/3-D CNN. The relevance loss is to measure the relationships between the semantics of the entire sentence and video content in the embedding space, while the coherence loss is to characterize the contextual relationships among the generated words in the sentence in LSTM. Both LSTM and visual-semantic embedding are jointly learnt by minimizing the two losses.

Datasets

The model was evaluated on the YouTube2Text (also known as MSVD) dataset.

  • Content: 1,970 YouTube snippets covering diverse daily activities.
  • Annotations: Approximately 40 English descriptions per video.
  • Split: 1,200 videos for training, 100 for validation, and 670 for testing.

Key Insights and Ideas

  • Global vs. Local: The "aha moment" is that local word prediction isn't enough for video understanding. By enforcing a global relationship in an embedding space, the model avoids common semantic errors like misidentifying the subject or the action.
  • 3D CNNs for Verbs: The inclusion of C3D features significantly improved verb prediction accuracy. This makes sense: 2D frames can identify a "horse," but you need temporal information to distinguish between "galloping," "walking," or "standing."
  • The Optimal Balance: The authors found that a $\lambda$ value of approximately 0.7 provided the best results, proving that while coherence is important, semantic relevance should carry significant weight in the training objective.
Figure 4. The effect of the tradeoff parameter λ measured by BLEU@N and METEOR.

Key Results

LSTM-E set new benchmarks on the YouTube2Text dataset, particularly when combining VGG and C3D features:

  • BLEU@4: Achieved 45.3%, a significant jump over the traditional LSTM baseline (31.2%).
  • METEOR: Reached 31.0%, outperforming then-current state-of-the-art methods like Soft-Attention (29.6%) and S2VT (29.8%).
  • SVO Accuracy: The model was notably better at predicting Subject-Verb-Object triplets. For instance, its verb prediction accuracy (29.85%) was much higher than the baseline LSTM (19.40%), leading to more truthful descriptions.

Limitations and Future Work

The authors identify two main areas for improvement:

  1. Temporal Modeling: While mean pooling is effective, it's a relatively simple way to aggregate video features. Future work could explore using RNNs to better represent the temporal evolution of the video itself before feeding it into the captioning stage.
  2. Data Scarcity: Deep learning models thrive on data. The authors suggest that video description generation could see even larger gains if more massive, high-quality labeled video-sentence datasets were available to train deeper architectures.

Summary updated

Abstract

Automatically describing video content with natural language is a fundamental challenge of multimedia. Recurrent Neural Networks (RNN), which models sequence dynamics, has attracted increasing attention on visual interpretation. However, most existing approaches generate a word locally with given previous words and the visual content, while the relationship between sentence semantics and visual content is not holistically exploited. As a result, the generated sentences may be contextually correct but the semantics (e.g., subjects, verbs or objects) are not true. This paper presents a novel unified framework, named Long Short-Term Memory with visual-semantic Embedding (LSTM-E), which can simultaneously explore the learning of LSTM and visual-semantic embedding. The former aims to locally maximize the probability of generating the next word given previous words and visual content, while the latter is to create a visual-semantic embedding space for enforcing the relationship between the semantics of the entire sentence and visual content. Our proposed LSTM-E consists of three components: a 2-D and/or 3-D deep convolutional neural networks for learning powerful video representation, a deep RNN for generating sentences, and a joint embedding model for exploring the relationships between visual content and sentence semantics. The experiments on YouTube2Text dataset show that our proposed LSTM-E achieves to-date the best reported performance in generating natural sentences: 45.3% and 31.0% in terms of BLEU@4 and METEOR, respectively. We also demonstrate that LSTM-E is superior in predicting Subject-Verb-Object (SVO) triplets to several state-of-the-art techniques.
BibTeX
@inproceedings{pan2016jointly,
  author = {Yingwei Pan and Tao Mei and Ting Yao and Houqiang Li and Yong Rui},
  title = {{Jointly Modeling Embedding and Translation to Bridge Video and Language}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2016},
  doi = {10.1109/cvpr.2016.497},
}
Figure