Multi-Level Multimodal Transformer Network for Multimodal Recipe Comprehension

Ao Liu, Shuai Yuan, Chenbin Zhang, Congjian Luo, Yaqing Liao, Kun Bai, Zenglin Xu

Abstract

Multimodal Machine Comprehension ($\rm M^3C$) has been a challenging task that requires understanding both language and vision, as well as their integration and interaction. For example, the RecipeQA challenge, which provides several $\rm M^3C$ tasks, requires deep neural models to understand textual instructions, images of different steps, as well as the logic orders of food cooking. To address this challenge, we propose a Multi-Level Multi-Modal Transformer (MLMM-Trans) framework to integrate and understand multiple textual instructions and multiple images. Our model can conduct intensive attention mechanism at multiple levels of objects (e.g., step level and passage-image level) for sequences of different modalities. Experiments have shown that our model can achieve the state-of-the-art results on the three multimodal tasks of RecipeQA.
BibTeX
@inproceedings{liu2020multi,
  author = {Ao Liu and Shuai Yuan and Chenbin Zhang and Congjian Luo and Yaqing Liao and Kun Bai and Zenglin Xu},
  title = {{Multi-Level Multimodal Transformer Network for Multimodal Recipe Comprehension}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  doi = {10.1145/3397271.3401247},
}

Loading related content...

Figure