Multi-modal Feature Fusion with Feature Attention for VATEX Captioning Challenge 2020

Ke Lin, Zhuoxin Gan, Liwei Wang

Abstract

This report describes our model for the VATEX Captioning Challenge 2020. To gather information from multiple domains, we extract motion, appearance, semantic, and audio features. We design a feature attention module to attend to different features during decoding. We apply two types of decoders, Top-down and X-LAN, and ensemble these models to achieve the final result. The proposed method outperforms the official baseline by a significant margin, achieving 76.0 CIDEr on the English private test set and 50.4 CIDEr on the Chinese private test set. We ranked 2nd on both the English and Chinese private test leaderboards.

Introduction

Video captioning is the challenging task of automatically generating natural language descriptions for video clips. Unlike static image captioning, videos are information-dense, containing temporal motion, background audio, and evolving semantic themes. Effectively capturing and fusing these disparate data types is critical for generating accurate and descriptive captions.

The authors address two primary hurdles in this domain:

  1. Information Overload: Effectively combining diverse data types (e.g., audio, motion, and regional appearance) without introducing noise.
  2. Exposure Bias: Addressing the discrepancy between training (where ground truth words are provided) and inference (where the model relies on its own previous predictions).

Methodology

The proposed architecture utilizes a sophisticated multi-modal feature extraction pipeline coupled with a dynamic attention mechanism.

Multi-modal Feature Extraction

Features are extracted from four distinct domains to provide a comprehensive representation of the video content:

  • Motion: Captured using I3D and SlowFast networks to model temporal dynamics.
  • Appearance: Global frame features are extracted via PNASNet, while regional "bottom-up" features are identified using Faster R-CNN. Additionally, ECO provides a hybrid 2D/3D fusion.
  • Audio: Soundscapes are converted into feature vectors using VGGish.
  • Semantics: A semantic prior is established by training a model to detect the 300 most frequent words (attributes) in the dataset.
Overview of the multi-modal feature extraction and attention-based decoding architecture.

Feature Attention and Decoding

To integrate these diverse inputs, a Feature Attention module calculates dynamic weights for each modality at every decoding time step. This allows the model to prioritize specific features—such as audio when a sound is prominent or motion during an action—based on the current context.

The system employs two state-of-the-art decoders:

  • Top-down: A two-layer GRU with standard attention.
  • X-LAN: Utilizes "X-Linear" attention to capture higher-order interactions between features.

The final output is an ensemble of these two architectures, leveraging the complementary information learned by each.

Multi-Stage Training Strategy

To mitigate exposure bias and optimize performance, the model undergoes a three-stage training process:

  1. Cross-Entropy Loss: Standard next-word prediction training.
  2. Word-level Oracle: A refinement stage to improve word selection.
  3. Self-Critical Policy Gradient: A Reinforcement Learning (RL) approach that directly optimizes the model for the CIDEr metric.

Results

The model was evaluated on the VATEX dataset, which includes over 41,250 video clips with multilingual descriptions. The proposed method significantly outperformed the official VATEX baselines.

Performance comparison of Top-down, X-LAN, and Ensemble models on the VATEX English validation and test sets.

As shown in the results, the ensemble approach achieved 76.0 CIDEr on the English private test set. For the Chinese private test set, the model achieved 50.4 CIDEr. These scores secured 2nd place globally for both tracks of the VATEX 2020 challenge.

Conclusion

This work demonstrates that a well-managed ensemble of diverse multi-modal features—including audio, motion, and semantic priors—can substantially improve video captioning performance. The use of a dynamic feature attention mechanism is crucial for effectively fusing these modalities. While the ensemble approach yields superior results, future research could focus on distilling this knowledge into more computationally efficient single-stream models to reduce the overhead of running multiple high-end networks.

Summary updated

Abstract

This report describes our model for VATEX Captioning Challenge 2020. First, to gather information from multiple domains, we extract motion, appearance, semantic and audio features. Then we design a feature attention module to attend on different feature when decoding. We apply two types of decoders, top-down and X-LAN and ensemble these models to get the final result. The proposed method outperforms official baseline with a significant gap. We achieve 76.0 CIDEr and 50.0 CIDEr on English and Chinese private test set. We rank 2nd on both English and Chinese private test leaderboard.
BibTeX
@misc{lin2020multi,
  author = {Ke Lin and Zhuoxin Gan and Liwei Wang},
  title = {{Multi-modal Feature Fusion with Feature Attention for VATEX Captioning Challenge 2020}},
  year = {2020},
  eprint = {2006.03315},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.03315},
}
Figure