Describing Videos using Multi-modal Fusion
Lists
Only you see this
Abstract
Describing videos with natural language is one of the ultimate goals of video understanding. Video records multi-modal information including image, motion, aural, speech and so on. MSR Video to Language Challenge provides a good chance to study multi-modality fusion in caption task. In this paper, we propose the multi-modal fusion encoder and integrate it with text sequence decoder into an end-to-end video caption framework. Features from visual, aural, speech and meta modalities are fused together to represent the video contents. Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs) are then used as the decoder to generate natural language sentences. Experimental results show the effectiveness of multi-modal fusion encoder trained in the end-to-end framework, which achieved top performance in both common metrics evaluation and human evaluation.
BibTeX
@inproceedings{jin2016describing,
author = {Qin Jin and Jia Chen and Shizhe Chen and Yifan Xiong and Alexander G. Hauptmann},
title = {{Describing Videos using Multi-modal Fusion}},
booktitle = {ACM Multimedia},
year = {2016},
doi = {10.1145/2964284.2984065},
}Loading related content...