Abstract
Cross-modal search—retrieving relevant images from text queries or vice versa—is a fundamental component of modern web engines. To achieve efficiency at scale, researchers utilize hashing to compress high-dimensional data into compact binary strings. However, existing methods often fail to bridge the "semantic gap" because they ignore implicit fine-grained relationships or rely on scarce labeled data. This paper introduces CMGCH (Self-Supervised Multi-Grained Multi-Modal Knowledge Graph Contrastive Hashing), a novel framework that constructs a multi-modal knowledge graph to capture complex inter- and intra-modal relationships. By fusing global coarse-grained features with local fine-grained entities through a multi-granularity contrastive learning approach, CMGCH achieves significant performance gains over state-of-the-art methods without requiring manual labels.
Introduction
The primary challenge in cross-modal retrieval is the inherent difference in data representation between images and text. Traditional hashing techniques typically focus on global features, which may miss subtle but critical semantic links. For instance, an image containing a "bike" in a "park" might be paired with text describing a "sunny day." Without modeling the implicit relationship between these entities, retrieval accuracy suffers. Furthermore, the high cost of manual annotation limits the scalability of supervised models. CMGCH addresses these issues by using self-supervised graph reasoning to uncover hidden semantic structures.
The CMGCH Framework
The architecture of CMGCH is built upon three primary pillars designed to extract and align features across different levels of granularity.
1. Multi-modal Knowledge Graph (MKG) Construction
The model first extracts visual entities (objects) from images using Faster R-CNN and word entities from text using BERT. These entities form the nodes of a Multimodal Knowledge Graph. Edges are established based on two criteria: inter-modal co-occurrence (linking objects to the words that describe them) and intra-modal semantic similarity (linking words via WordNet or objects via visual similarity). This structure allows the model to reason about relationships that are not explicitly stated in the raw data.
2. Cross-modal Graph Contrastive Attention Network (CGCAN)
Using Graph Attention Networks (GAT), CMGCH performs reasoning over the MKG. It employs a contrastive learning objective (InfoNCE loss) to ensure that related entities across different modalities are pulled closer in the feature space, while unrelated ones are pushed apart. This process refines the local embeddings by considering their neighborhood context within the graph.
3. Multi-granularity Contrastive Embedding
To create a robust final representation, the model fuses global coarse-grained features (representing the entire image or sentence) with the local fine-grained features (derived from the graph entities). A multi-head attention mechanism is used to enhance these embeddings, ensuring the resulting hash codes are both discriminative and semantically consistent across modalities.
Experimental Evaluation
CMGCH was evaluated on two gold-standard benchmarks: MSCOCO (123,000+ images) and Flickr30k (31,000 images). The model was compared against 16 state-of-the-art methods, including both unsupervised (e.g., UCCH, JDSH) and supervised (e.g., DLFH) approaches. To facilitate self-supervised training, the authors utilized a momentum encoder and a dynamic queue, allowing the model to contrast samples against a large dictionary of negatives.
Key Results
The quantitative improvements demonstrated by CMGCH are substantial:
- Accuracy: On MSCOCO, CMGCH achieved a 16% to 32% increase in Mean Average Precision (MAP) across various bit lengths compared to previous benchmarks.
- Retrieval Performance: On Flickr30k, the model showed a dramatic 159% improvement in Recall@1 for Image-to-Text tasks at 64 bits over the next best competitor.
- Efficiency: Despite the complexity of graph reasoning, inference remains fast at approximately 0.018 seconds, making it suitable for real-time applications.
Conclusion
CMGCH demonstrates that self-supervised structural information, when modeled via multi-modal knowledge graphs, can outperform traditional supervised methods. By integrating multi-granularity features, the model effectively bridges the semantic gap. Future work may involve scaling graph reasoning to billion-scale datasets and expanding the framework to include additional modalities such as video and audio.
Summary updated