Region-based Cross-modal Retrieval
Lists
Only you see this
Abstract
Cross-modal retrieval aims to identify relevant information from different modalities, such as image and text. Existing works build a coarse relationship between whole image and whole text. However, the retrieval in fine-grained elements is more necessary in research and has many practical applications, such as explainable and interactive retrieval. In this paper, we propose a region-based cross-modal retrieval task that focuses on finding the semantic match between fine-grained parts of text and image, e.g., sentence in paragraph and region in image. The challenge of this task is to enhance fine-grained representation with contextual knowledge. To this end, we propose a Context-Aware Region Retrieval (CARR) model. It utilizes two pre-trained models as backbones, Faster RCNN for image and BERT for text, and transformer-based encoders to obtain the contextual aware representations of two modalities. We conduct experiments on Visual Genome and Localized Narratives datasets, and the experimental results demonstrate that the proposed model outperforms the baseline methods.
BibTeX
@inproceedings{hou2022region,
author = {Dan Yang Hou and Liang Pang and Yanyan Lan and Huawei Shen and Xueqi Cheng},
title = {{Region-based Cross-modal Retrieval}},
booktitle = {IJCNN},
year = {2022},
doi = {10.1109/ijcnn55064.2022.9892139},
}Loading related content...