Dual Semantic Fusion Hashing for Multi-Label Cross-Modal Retrieval

Kaiming Liu, Yunhong Gong, Yu Cao, Zhenwen Ren, Dezhong Peng

Abstract: Cross-modal retrieval (CMR) is a fundamental task in modern information systems, enabling searches across disparate data modalities such as images and text. To achieve scalability, Cross-Modal Hashing (CMH) maps high-dimensional data into compact binary codes. However, existing CMH methods often rely on coarse instance-level similarity, failing to capture the complex semantic distributions inherent in multi-label datasets. We present Dual Semantic Fusion Hashing (DSFH), a novel framework that integrates traditional instance-level labels with fine-grained cluster-level semantics. By leveraging K-Means clustering to identify latent semantic structures, DSFH guides hash code learning through a dual-layer supervision mechanism. Our evaluation across three benchmark datasets—MIRFlickr-25K, IAPR-TC12, and NUS-WIDE—demonstrates that DSFH significantly outperforms 16 state-of-the-art methods in retrieval accuracy while maintaining linear computational complexity.

Introduction

The explosion of multi-modal big data has necessitated efficient retrieval techniques. Cross-modal hashing (CMH) has become a standard approach due to its low storage requirements and high query speeds. Despite its success, a significant challenge remains in handling multi-label data. Most current methods utilize a binary similarity matrix based on shared labels, which treats all label overlaps as equally significant. This "coarse-grained" approach ignores the varying densities and categorical relationships within the data. For example, a common label shared by many instances provides less discriminative information than a rare label shared by a few. DSFH addresses this by incorporating "fine-grained" cluster-level semantics into the hashing process.

The DSFH Framework

The DSFH framework operates through a multi-stage process designed to align different modalities while preserving their unique semantic nuances.

Overview of the DSFH framework, illustrating the transition from modal-specific representations to consensus hash codes guided by dual semantic fusion.

The pipeline begins with Modality-Specific Representation, where features are learned independently for each modality to preserve their inherent characteristics. Subsequently, Fine-Grained Semantic Construction is performed by applying K-Means clustering to the label matrix. This identifies clusters of labels that frequently co-occur, forming a cluster-level similarity matrix ($S_C$). Finally, Dual Semantic Fusion combines the instance-level ($S_I$) and cluster-level ($S_C$) matrices to supervise the learning of consensus hash codes. This dual-layer approach ensures that the resulting binary codes respect both specific label assignments and broader semantic structures.

Pull and Push Dynamics

A key conceptual contribution of DSFH is the refinement of semantic neighborhoods through "Pull and Push" dynamics. In traditional hashing, items sharing any label are pulled together with equal force. DSFH refines this by considering cluster density.

The "Pull and Push" mechanism in DSFH, showing how cluster-level semantics refine similarity by pulling related dense clusters closer and pushing irrelevant ones further away.

As illustrated in Figure 1, the model can "pull" related but distinct labels closer if they belong to a dense semantic neighborhood, while "pushing" irrelevant ones further away more effectively. This results in a more discriminative Hamming space, where the distance between hash codes better reflects true semantic similarity.

Experimental Evaluation

We evaluated DSFH on three gold-standard benchmarks: MIRFlickr-25K, IAPR-TC12, and NUS-WIDE. The framework was compared against 11 shallow and 5 deep CMH methods. DSFH consistently achieved the highest Mean Average Precision (mAP) across all datasets and bit lengths (8 to 64 bits). Notably, in image-to-text tasks, DSFH showed improvements of up to 3.76% over the next best baseline. Furthermore, the DSFH-cnn variant, which utilizes deep features, outperformed specialized end-to-end deep hashing models, highlighting the effectiveness of the dual-semantic fusion strategy.

Conclusion

Dual Semantic Fusion Hashing provides a powerful and efficient approach to multi-label cross-modal retrieval. By bridging the gap between coarse labels and fine-grained semantic clusters, DSFH produces hash codes with superior discriminative power. The linear time complexity of the optimization process ensures that the method remains scalable for large-scale applications. Future research will focus on integrating this dual-semantic logic into end-to-end deep learning architectures to further push the boundaries of cross-modal search performance.

Summary updated

BibTeX
@inproceedings{liu2024dual,
  author = {Kaiming Liu and Yunhong Gong and Yu Cao and Zhenwen Ren and Dezhong Peng},
  title = {{Dual Semantic Fusion Hashing for Multi-Label Cross-Modal Retrieval}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=rGROSCT8Xx},
}
Figure