Introduction
Generating natural language descriptions for video remains a significant challenge in computer vision. A successful system must go beyond simple object recognition to understand saliency—identifying which elements of a complex scene are most relevant to a human observer. Rohrbach et al. (CVPR 2015) introduced the MPII Movie Description (MPII-MD) dataset to address this by leveraging professional Audio Descriptions (AD).
The Challenge of Video Description Data
Prior to MPII-MD, research was limited by small or domain-specific datasets. Many researchers relied on movie scripts, which are problematic because they are written pre-production. Scripts often include non-visual information, internal monologues, or scenes that are ultimately altered. In contrast, Audio Descriptions are created post-production for the visually impaired, providing a precise, temporally aligned account of on-screen action.
The MPII-MD Dataset
The MPII-MD dataset consists of over 68,000 sentences aligned with video snippets from 94 high-definition movies, totaling approximately 73.6 hours of video. Unlike previous resources, it provides a diverse, open-domain collection of realistic settings.
Methodology
The researchers employed a semi-automatic pipeline for dataset construction:
- AD Extraction: Narrator audio was isolated from Blu-ray tracks using spectrogram-based segmentation.
- Transcription and Alignment: Crowdsourced workers transcribed the audio, which was then manually aligned to specific video frames.
- Semantic Parsing: Sentences were decomposed into structured [Subject, Verb, Object, Location] tuples using tools like WordNet and VerbNet to facilitate machine learning.
- Benchmarking: The team evaluated various models, including Nearest Neighbor retrieval and a Statistical Machine Translation (SMT) approach that maps visual features to semantic tuples.
Key Insights: Saliency and Representation
A primary finding is that AD is superior to scripts for training vision models. Human evaluators preferred AD descriptions 66.1% of the time, noting their higher relevance to the visual content. AD naturally filters for saliency; because narrators have limited time between dialogues, they focus only on essential visual information.
Furthermore, the study demonstrated that using intermediate semantic tuples (SVO) as a bridge between pixels and prose leads to more accurate and grammatically sound descriptions than direct translation.
Results and Performance
In benchmarking, the SMT approach using semantic labels outperformed other automated methods. On the TACoS dataset, the semantic parsing approach achieved a BLEU@4 score of 24.0, nearing the performance of models trained on manual annotations (26.9). For feature extraction, "Hybrid" features—combining object and scene recognition—proved more effective than motion-based features for retrieval tasks.
Conclusion and Future Directions
While MPII-MD represents a major advancement, scaling remains a challenge due to the need for manual alignment. Future research must address the "open domain" gap, moving from basic action descriptions to understanding complex human interactions and long-term plot dependencies across multiple video snippets.
Summary updated