Dual Semantic Fusion Hashing for Multi-Label Cross-Modal Retrieval

Kaiming Liu, Yunhong Gong, Yu Cao, Zhenwen Ren, Dezhong Peng, Yuan Sun

Abstract

Cross-modal hashing (CMH) has been widely used for multi-modal retrieval tasks due to its low storage cost and fast query speed. Although existing CMH methods achieve promising performance, most of them mainly rely on coarse-grained supervision information (i.e., pairwise similarity matrix) to measure the semantic similarities between all instances, ignoring the impact of multi-label distribution. To address this issue, we construct fine-grained semantic similarity to explore the cluster-level semantic relationships between multi-label data, and propose a new dual semantic fusion hashing (DSFH) for multi-label cross-modal retrieval. Specifically, we first learn the modal-specific representation and consensus hash codes, thereby merging the specificity with consistency. Then, we fuse the coarse-grained and fine-grained semantics to mine multiple-level semantic relationships, thereby enhancing hash codes discrimination. Extensive experiments on three benchmarks demonstrate the superior performance of our DSFH compared with 16 state-of-the-art methods.

Introduction

In the era of massive multi-modal data, efficient retrieval across different media—such as finding relevant text for an image—is a critical challenge. Cross-Modal Hashing (CMH) has emerged as a leading solution, compressing high-dimensional data into compact binary codes to enable rapid search in Hamming space. However, a significant hurdle remains in handling multi-label data, where instances are associated with multiple semantic tags.

The Challenge of Multi-Label Data

Most supervised CMH methods bridge the "heterogeneity gap" using a pairwise similarity matrix. In this coarse-grained approach, two items are considered similar if they share at least one label. This binary view fails to capture the nuanced semantic density of label distributions. For example, if an instance A shares a label with a large cluster B and a small cluster C, traditional models treat A as equally similar to both. Intuitively, A should be more closely aligned with the larger semantic cluster to improve retrieval relevance.

Illustration of the transition from coarse-grained similarity to fine-grained cluster-level relationships.

The DSFH Framework

To address these limitations, the Dual Semantic Fusion Hashing (DSFH) framework introduces a multi-level approach to semantic modeling. The architecture consists of three primary stages:

  1. Modality-Specific Representation: The model learns representations for each modality (e.g., images and text) that preserve unique characteristics while ensuring cross-modal consistency.
  2. Dual Semantic Construction: DSFH constructs two distinct similarity matrices:
  3. Instance-level ($S_I$): Captures standard coarse-grained relationships.
  4. Cluster-level ($S_C$): Captures fine-grained relationships by applying K-Means clustering to label vectors, grouping instances with similar semantic distributions.
  5. Dual Semantic Fusion: These views are integrated into a joint semantic matrix ($S_J$) to guide the generation of highly discriminative consensus hash codes.
The overall architecture of the Dual Semantic Fusion Hashing (DSFH) framework, showing modal-specific learning and dual semantic fusion.

Experimental Evaluation

The researchers evaluated DSFH on three major benchmarks: MIRFlickr-25K, IAPR-TC12, and NUS-WIDE. Compared against 16 state-of-the-art methods (including 11 shallow and 5 deep hashing models), DSFH demonstrated superior performance:

  • Accuracy: DSFH achieved the highest Mean Average Precision (mAP) across all tasks and bit lengths. On MIRFlickr-25K, it improved Image-to-Text retrieval by up to 3.76% over the best baseline.
  • Efficiency: The model maintains linear computational complexity $O(n)$, making it scalable for large datasets.
  • Robustness: A deep-feature variant (DSFH-cnn) outperformed dedicated end-to-end deep hashing networks, highlighting the effectiveness of the dual semantic fusion logic.

Conclusion

DSFH advances cross-modal retrieval by moving beyond simple pairwise similarity to embrace the structural information hidden in multi-label distributions. By fusing coarse-grained and fine-grained semantics, it produces more discriminative hash codes without sacrificing computational efficiency. Future work may explore integrating this dual semantic logic directly into end-to-end deep learning architectures or utilizing more complex clustering techniques to handle non-linear label manifolds.

Summary updated

Abstract

Cross-modal hashing (CMH) has been widely used for multi-modal retrieval tasks due to its low storage cost and fast query speed. Although existing CMH methods achieve promising performance, most of them mainly rely on coarse-grained supervision information (\ie pairwise similarity matrix) to measure the semantic similarities between all instances, ignoring the impact of multi-label distribution. To address this issue, we construct fine-grained semantic similarity to explore the cluster-level semantic relationships between multi-label data, and propose a new dual semantic fusion hashing (DSFH) for multi-label cross-modal retrieval. Specifically, we first learn the modal-specific representation and consensus hash codes, thereby merging the specificity with consistency. Then, we fuse the coarse-grained and fine-grained semantics to mine multiple-level semantic relationships, thereby enhancing hash codes discrimination. Extensive experiments on three benchmarks demonstrate the superior performance of our DSFH compared with 16 state-of-the-art methods.
BibTeX
@inproceedings{liu2024dual,
  author = {Kaiming Liu and Yunhong Gong and Yu Cao and Zhenwen Ren and Dezhong Peng and Yuan Sun},
  title = {{Dual Semantic Fusion Hashing for Multi-Label Cross-Modal Retrieval}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=rGROSCT8Xx},
}
Figure