Describing Videos using Multi-modal Fusion

Qin Jin, Jia Chen, Shizhe Chen, Yifan Xiong, Alexander G. Hauptmann

Abstract

Describing videos with natural language is one of the ultimate goals of video understanding. Video records multi-modal information including image, motion, aural, speech and so on. MSR Video to Language Challenge provides a good chance to study multi-modality fusion in caption task. In this paper, we propose the multi-modal fusion encoder and integrate it with text sequence decoder into an end-to-end video caption framework. Features from visual, aural, speech and meta modalities are fused together to represent the video contents. Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs) are then used as the decoder to generate natural language sentences. Experimental results show the effectiveness of multi-modal fusion encoder trained in the end-to-end framework, which achieved top performance in both common metrics evaluation and human evaluation.
BibTeX
@inproceedings{jin2016describing,
  author = {Qin Jin and Jia Chen and Shizhe Chen and Yifan Xiong and Alexander G. Hauptmann},
  title = {{Describing Videos using Multi-modal Fusion}},
  booktitle = {ACM Multimedia},
  year = {2016},
  doi = {10.1145/2964284.2984065},
}

Loading related content...

Figure