Multi-granular Semantic Mining for Composed Image Retrieval

Xiaotong Chen, Shikui Wei, Gangjian Zhang, Yao Zhao

Abstract

Composed Image Retrieval(CIR) aims to model users’ query intention with multiple modalities and retrieve the desired images from a large image corpus. The biggest challenge is how to effectively integrate the semantic information between two different modalities. A popular solution is to design attention-based modules to extract the query embedding in a coarse manner, which leads to certain confusion about search intention. To address this problem, we propose a new method for query integration, which is composed of two key modules, i.e., Multi-granular Subspace Fusion (MSF) and Residual Regression (RR) constraint. Specifically, MSF focuses on mining cross-modal semantic dependency between reference image regions and modification text pieces in multi-granular subspaces, which can construct an implicit, holistic semantic relationship in a fine manner. And RR constraint pushes the visual-text semantic alignment under specific supervision. Extensive experiments on three prevalent datasets demonstrate the state-of-the-art performance of our method.
BibTeX
@inproceedings{chen2024multi,
  author = {Xiaotong Chen and Shikui Wei and Gangjian Zhang and Yao Zhao},
  title = {{Multi-granular Semantic Mining for Composed Image Retrieval}},
  booktitle = {ICME},
  year = {2024},
  doi = {10.1109/icme57554.2024.10687702},
}

Loading related content...

Figure