Self-Supervised Multi-Modal Knowledge Graph Contrastive Hashing for Cross-Modal Search

Meiyu Liang, Junping Du, Zhengyang Liang, Yongwang Xing, Wei Huang, Zhe Xue

Abstract

Cross-modal search—retrieving relevant images from text queries or vice versa—is a fundamental component of modern web engines. To achieve efficiency at scale, researchers utilize hashing to compress high-dimensional data into compact binary strings. However, existing methods often fail to bridge the "semantic gap" because they ignore implicit fine-grained relationships or rely on scarce labeled data. This paper introduces CMGCH (Self-Supervised Multi-Grained Multi-Modal Knowledge Graph Contrastive Hashing), a novel framework that constructs a multi-modal knowledge graph to capture complex inter- and intra-modal relationships. By fusing global coarse-grained features with local fine-grained entities through a multi-granularity contrastive learning approach, CMGCH achieves significant performance gains over state-of-the-art methods without requiring manual labels.

Introduction

The primary challenge in cross-modal retrieval is the inherent difference in data representation between images and text. Traditional hashing techniques typically focus on global features, which may miss subtle but critical semantic links. For instance, an image containing a "bike" in a "park" might be paired with text describing a "sunny day." Without modeling the implicit relationship between these entities, retrieval accuracy suffers. Furthermore, the high cost of manual annotation limits the scalability of supervised models. CMGCH addresses these issues by using self-supervised graph reasoning to uncover hidden semantic structures.

The CMGCH Framework

The architecture of CMGCH is built upon three primary pillars designed to extract and align features across different levels of granularity.

1. Multi-modal Knowledge Graph (MKG) Construction

The model first extracts visual entities (objects) from images using Faster R-CNN and word entities from text using BERT. These entities form the nodes of a Multimodal Knowledge Graph. Edges are established based on two criteria: inter-modal co-occurrence (linking objects to the words that describe them) and intra-modal semantic similarity (linking words via WordNet or objects via visual similarity). This structure allows the model to reason about relationships that are not explicitly stated in the raw data.

Figure 1: Construction of the Multimodal Knowledge Graph (MKG), showing intra-modal semantic relations and inter-modal co-occurrence relations.

2. Cross-modal Graph Contrastive Attention Network (CGCAN)

Using Graph Attention Networks (GAT), CMGCH performs reasoning over the MKG. It employs a contrastive learning objective (InfoNCE loss) to ensure that related entities across different modalities are pulled closer in the feature space, while unrelated ones are pushed apart. This process refines the local embeddings by considering their neighborhood context within the graph.

3. Multi-granularity Contrastive Embedding

To create a robust final representation, the model fuses global coarse-grained features (representing the entire image or sentence) with the local fine-grained features (derived from the graph entities). A multi-head attention mechanism is used to enhance these embeddings, ensuring the resulting hash codes are both discriminative and semantically consistent across modalities.

Figure 2: Overview of the CMGCH architecture, illustrating the flow from raw data to hash codes via multi-granularity embedding and contrastive learning.

Experimental Evaluation

CMGCH was evaluated on two gold-standard benchmarks: MSCOCO (123,000+ images) and Flickr30k (31,000 images). The model was compared against 16 state-of-the-art methods, including both unsupervised (e.g., UCCH, JDSH) and supervised (e.g., DLFH) approaches. To facilitate self-supervised training, the authors utilized a momentum encoder and a dynamic queue, allowing the model to contrast samples against a large dictionary of negatives.

Key Results

The quantitative improvements demonstrated by CMGCH are substantial:

  • Accuracy: On MSCOCO, CMGCH achieved a 16% to 32% increase in Mean Average Precision (MAP) across various bit lengths compared to previous benchmarks.
  • Retrieval Performance: On Flickr30k, the model showed a dramatic 159% improvement in Recall@1 for Image-to-Text tasks at 64 bits over the next best competitor.
  • Efficiency: Despite the complexity of graph reasoning, inference remains fast at approximately 0.018 seconds, making it suitable for real-time applications.

Conclusion

CMGCH demonstrates that self-supervised structural information, when modeled via multi-modal knowledge graphs, can outperform traditional supervised methods. By integrating multi-granularity features, the model effectively bridges the semantic gap. Future work may involve scaling graph reasoning to billion-scale datasets and expanding the framework to include additional modalities such as video and audio.

Summary updated

Abstract

Deep cross-modal hashing technology provides an effective and efficient cross-modal unified representation learning solution for cross-modal search. However, the existing methods neglect the implicit fine-grained multimodal knowledge relations between these modalities such as when the image contains information that is not directly described in the text. To tackle this problem, we propose a novel self-supervised multi-grained multi-modal knowledge graph contrastive hashing method for cross-modal search (CMGCH). Firstly, in order to capture implicit fine-grained cross-modal semantic associations, a multi-modal knowledge graph is constructed, which represents the implicit multimodal knowledge relations between the image and text as inter-modal and intra-modal semantic associations. Secondly, a cross-modal graph contrastive attention network is proposed to reason on the multi-modal knowledge graph to sufficiently learn the implicit fine-grained inter-modal and intra-modal knowledge relations. Thirdly, a cross-modal multi-granularity contrastive embedding learning mechanism is proposed, which fuses the global coarse-grained and local fine-grained embeddings by multihead attention mechanism for inter-modal and intra-modal contrastive learning, so as to enhance the cross-modal unified representations with stronger discriminativeness and semantic consistency preserving power. With the joint training of intra-modal and inter-modal contrast, the invariant and modal-specific information of different modalities can be maintained in the final unified cross-modal unified hash space. Extensive experiments on several cross-modal benchmark datasets demonstrate that the proposed CMGCH outperforms the state-of the-art methods.
BibTeX
@inproceedings{liang2024self,
  author = {Meiyu Liang and Junping Du and Zhengyang Liang and Yongwang Xing and Wei Huang and Zhe Xue},
  title = {{Self-Supervised Multi-Modal Knowledge Graph Contrastive Hashing for Cross-Modal Search}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i12.29280},
}
Figure