Mining on Heterogeneous Manifolds for Zero-Shot Cross-Modal Image Retrieval
Lists
Only you see this
Abstract
Most recent approaches for the zero-shot cross-modal image retrieval map images from different modalities into a uniform feature space to exploit their relevance by using a pre-trained model. Based on the observation that manifolds of zero-shot images are usually deformed and incomplete, we argue that the manifolds of unseen classes are inevitably distorted during the training of a two-stream model that simply maps images from different modalities into a uniform space. This issue directly leads to poor cross-modal retrieval performance. We propose a bi-directional random walk scheme to mining more reliable relationships between images by traversing heterogeneous manifolds in the feature space of each modality. Our proposed method benefits from intra-modal distributions to alleviate the interference caused by noisy similarities in the cross-modal feature space. As a result, we achieved great improvement in the performance of the thermal v.s. visible image retrieval task. The code of this paper: https://github.com/fyang93/cross-modal-retrieval
BibTeX
@inproceedings{yang2020mining,
author = {Fan Yang and Zheng Wang and Jing Xiao and Shin'ichi Satoh},
title = {{Mining on Heterogeneous Manifolds for Zero-Shot Cross-Modal Image Retrieval}},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2020},
doi = {10.1609/aaai.v34i07.6949},
}Loading related content...