A Reconstruction-Based Visual-Acoustic-Semantic Embedding Method for Speech-Image Retrieval

Wenlong Cheng, Wei Tang, Yan Huang, Yiwen Luo, Liang Wang

Abstract

Speech-image retrieval aims at learning the relevance between image and speech.Prior approaches are mainly based on bi-modal contrastive learning, which can not alleviate the cross-modal heterogeneous issue between visual and acoustic modalities well. To address this issue, we propose a visual-acoustic-semantic embedding (VASE) method. First, we propose a tri-modal ranking loss by taking advantage of semantic information corresponding to the acoustic data, which introduces the auxiliary alignment to enhance the alignment between image and speech. Second, we introduce a cycle-consistency loss based on feature reconstruction. It can further alleviate the heterogeneous issue between different data modalities ( e.g. , visual-acoustic, visual-textual and acoustic-textual). Extensive experiments have demonstrated the effectiveness of our proposed method. In addition, our VASE model achieves state-of-the-art performance on the speech-image retrieval task on the Flickr8K [Harwath and Glass, 2015]s and Places [Harwath et al. , 2018] datasets.
BibTeX
@article{cheng2023reconstruction,
  author = {Wenlong Cheng and Wei Tang and Yan Huang and Yiwen Luo and Liang Wang},
  title = {{A Reconstruction-Based Visual-Acoustic-Semantic Embedding Method for Speech-Image Retrieval}},
  journal = {IEEE Trans. Multimedia},
  year = {2023},
  doi = {10.1109/tmm.2022.3171090},
}

Loading related content...

Figure