Abstract
This report describes our model for the VATEX Captioning Challenge 2020. To gather information from multiple domains, we extract motion, appearance, semantic, and audio features. We design a feature attention module to attend to different features during decoding. We apply two types of decoders, Top-down and X-LAN, and ensemble these models to achieve the final result. The proposed method outperforms the official baseline by a significant margin, achieving 76.0 CIDEr on the English private test set and 50.4 CIDEr on the Chinese private test set. We ranked 2nd on both the English and Chinese private test leaderboards.
Introduction
Video captioning is the challenging task of automatically generating natural language descriptions for video clips. Unlike static image captioning, videos are information-dense, containing temporal motion, background audio, and evolving semantic themes. Effectively capturing and fusing these disparate data types is critical for generating accurate and descriptive captions.
The authors address two primary hurdles in this domain:
- Information Overload: Effectively combining diverse data types (e.g., audio, motion, and regional appearance) without introducing noise.
- Exposure Bias: Addressing the discrepancy between training (where ground truth words are provided) and inference (where the model relies on its own previous predictions).
Methodology
The proposed architecture utilizes a sophisticated multi-modal feature extraction pipeline coupled with a dynamic attention mechanism.
Multi-modal Feature Extraction
Features are extracted from four distinct domains to provide a comprehensive representation of the video content:
- Motion: Captured using I3D and SlowFast networks to model temporal dynamics.
- Appearance: Global frame features are extracted via PNASNet, while regional "bottom-up" features are identified using Faster R-CNN. Additionally, ECO provides a hybrid 2D/3D fusion.
- Audio: Soundscapes are converted into feature vectors using VGGish.
- Semantics: A semantic prior is established by training a model to detect the 300 most frequent words (attributes) in the dataset.
Feature Attention and Decoding
To integrate these diverse inputs, a Feature Attention module calculates dynamic weights for each modality at every decoding time step. This allows the model to prioritize specific features—such as audio when a sound is prominent or motion during an action—based on the current context.
The system employs two state-of-the-art decoders:
- Top-down: A two-layer GRU with standard attention.
- X-LAN: Utilizes "X-Linear" attention to capture higher-order interactions between features.
The final output is an ensemble of these two architectures, leveraging the complementary information learned by each.
Multi-Stage Training Strategy
To mitigate exposure bias and optimize performance, the model undergoes a three-stage training process:
- Cross-Entropy Loss: Standard next-word prediction training.
- Word-level Oracle: A refinement stage to improve word selection.
- Self-Critical Policy Gradient: A Reinforcement Learning (RL) approach that directly optimizes the model for the CIDEr metric.
Results
The model was evaluated on the VATEX dataset, which includes over 41,250 video clips with multilingual descriptions. The proposed method significantly outperformed the official VATEX baselines.
As shown in the results, the ensemble approach achieved 76.0 CIDEr on the English private test set. For the Chinese private test set, the model achieved 50.4 CIDEr. These scores secured 2nd place globally for both tracks of the VATEX 2020 challenge.
Conclusion
This work demonstrates that a well-managed ensemble of diverse multi-modal features—including audio, motion, and semantic priors—can substantially improve video captioning performance. The use of a dynamic feature attention mechanism is crucial for effectively fusing these modalities. While the ensemble approach yields superior results, future research could focus on distilling this knowledge into more computationally efficient single-stream models to reduce the overhead of running multiple high-end networks.
Summary updated