Multi-Modal Deep Analysis for Multimedia
Lists
Only you see this
Abstract
With the rapid development of Internet and multimedia services in the past decade, a huge amount of user-generated and service provider-generated multimedia data become available. These data are heterogeneous and multi-modal in nature, imposing great challenges for processing and analyzing them. Multi-modal data consist of a mixture of various types of data from different modalities such as texts, images, videos, audios etc. In this article, we present a deep and comprehensive overview for multi-modal analysis in multimedia. We introduce two scientific research problems, data-driven correlational representation and knowledge-guided fusion for multimedia analysis. To address the two scientific problems, we investigate them from the following aspects: 1) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multi-modal correlational representation</i> : multi-modal fusion of data across different modalities, and 2) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multi-modal data and knowledge fusion</i> : multi-modal fusion of data with domain knowledge. More specifically, on data-driven correlational representation, we highlight three important categories of methods, such as multi-modal deep representation, multi-modal transfer learning, and multi-modal hashing. On knowledge-guided fusion, we discuss the approaches for fusing knowledge with data and four exemplar applications that require various kinds of domain knowledge, including multi-modal visual question answering, multi-modal video summarization, multi-modal visual pattern mining and multi-modal recommendation. Finally, we bring forward our insights and future research directions.
BibTeX
@article{zhu2019multi,
author = {Wenwu Zhu and Xin Wang and Hongzhi Li},
title = {{Multi-Modal Deep Analysis for Multimedia}},
journal = {IEEE Trans. Circuits Syst. Video Technol.},
year = {2019},
doi = {10.1109/tcsvt.2019.2940647},
}| Cross-media analysis and reasoning: advances and directions | Frontiers Inf. Technol. Electron. Eng · 2017 |
| Multimodal Intelligence: Representation Learning, Information Fusion, and Applications | arXiv · 2019 |
| Cross-Modal Representation | 2020 |
| A Review on Methods and Applications in Multimodal Deep Learning | arXiv · 2022 |
| Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications | arXiv · 2023 |
| A Survey of Multi-View Representation Learning | arXiv · 2016 |