Multi-Level Multimodal Transformer Network for Multimodal Recipe Comprehension
Lists
Only you see this
Abstract
Multimodal Machine Comprehension ($\rm M^3C$) has been a challenging task that requires understanding both language and vision, as well as their integration and interaction. For example, the RecipeQA challenge, which provides several $\rm M^3C$ tasks, requires deep neural models to understand textual instructions, images of different steps, as well as the logic orders of food cooking. To address this challenge, we propose a Multi-Level Multi-Modal Transformer (MLMM-Trans) framework to integrate and understand multiple textual instructions and multiple images. Our model can conduct intensive attention mechanism at multiple levels of objects (e.g., step level and passage-image level) for sequences of different modalities. Experiments have shown that our model can achieve the state-of-the-art results on the three multimodal tasks of RecipeQA.
BibTeX
@inproceedings{liu2020multi,
author = {Ao Liu and Shuai Yuan and Chenbin Zhang and Congjian Luo and Yaqing Liao and Kun Bai and Zenglin Xu},
title = {{Multi-Level Multimodal Transformer Network for Multimodal Recipe Comprehension}},
booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
year = {2020},
doi = {10.1145/3397271.3401247},
}Loading related content...