Multilateral Semantic Relations Modeling for Image Text Retrieval
Lists
Only you see this
Abstract
Image-text retrieval is a fundamental task to bridge vision and language by exploiting various strategies to finegrained alignment between regions and words. This is still tough mainly because of one-to-many correspondence, where a set of matches from another modality can be accessed by a random query. While existing solutions to this problem including multi-point mapping, probabilistic distribution, and geometric embedding have made promising progress, one-to-many correspondence is still under-explored. In this work, we develop a Multilateral Semantic Relations Modeling (termed MSRM) for image-text retrieval to capture the one-to-many correspondence between multiple samples and a given query via hypergraph modeling. Specifically, a given query is first mapped as a probabilistic embedding to learn its true semantic distribution based on Mahalanobis distance. Then each candidate instance in a mini-batch is regarded as a hypergraph node with its mean semantics while a Gaussian query is modeled as a hyperedge to capture the semantic correlations beyond the pair between candidate points and the query. Comprehensive experimental results on two widely used datasets demonstrate that our MSRM method can outper-form state-of-the-art methods in the settlement of multiple matches while still maintaining the comparable performance of instance-level matching.
BibTeX
@inproceedings{wang2023multilateral,
author = {Zheng Wang and Zhenwei Gao and Kangshuai Guo and Yang Yang and Xiaoming Wang and Heng Tao Shen},
title = {{Multilateral Semantic Relations Modeling for Image Text Retrieval}},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2023},
url = {https://openreview.net/forum?id=IKW3nPcJZRG},
}Loading related content...