Graph Convolutional Network Hashing for Cross-Modal Retrieval

Ruiqing Xu, Chao Li, Junchi Yan, Cheng Deng, Xianglong LiuIJCAI 2019 · OpenReview

Abstract

Deep network based cross-modal retrieval has recently made significant progress. However, bridging the modality gap to further enhance retrieval accuracy remains a crucial bottleneck. In this paper, we propose a Graph Convolutional Hashing (GCH) approach, which learns modality-unified binary codes via an affinity graph. An end-to-end deep architecture is constructed with three main components: a semantic encoder module, two feature encoding networks, and a graph convolutional network (GCN). We design a semantic encoder as a teacher module to guide the feature encoding process (student module) for semantic information exploitation. Furthermore, GCN is utilized to explore the inherent similarity structure among data points, which helps to generate discriminative hash codes. Extensive experiments on three benchmark datasets demonstrate that the proposed GCH outperforms state-of-the-art methods.

Introduction

Cross-modal retrieval is a cornerstone of modern search—enabling tasks like searching for specific videos using text descriptions. To make this process fast and memory-efficient, researchers utilize hashing to convert high-dimensional data (like pixels or word vectors) into short binary codes. However, two major hurdles remain: the modality gap, where images and text are fundamentally different and difficult to bridge into a single Hamming space, and the neglect of data relationships, where most methods treat data points as independent islands rather than exploiting their shared semantic structures.

Methodology

The proposed Graph Convolutional Hashing (GCH) architecture addresses these issues through a three-pronged end-to-end framework.

The proposed GCH architecture featuring a teacher-student framework and a GCN-based fusion module.
  • Semantic Encoder (The Teacher): This module takes ground-truth labels and learns a high-level semantic representation. It acts as a "teacher," providing guidance to the other networks on ideal semantic relationships.
  • Feature Encoding Networks (The Students): Two separate networks—a Convolutional Neural Network (CNN) for images and a Multi-Layer Perceptron (MLP) for text—learn to extract features. They are trained to mimic the teacher module, ensuring the extracted features are rich with semantic information.
  • GCN-based Fusion Module: This core innovation fuses image and text features using a self-attention mechanism before feeding them into a Graph Convolutional Network (GCN). The GCN uses an adjacency matrix built from labels to let neighboring data points interact, refining features based on their structural context. This creates a "beacon feature" that pulls similar items closer together in the final hash space.

Experimental Results

The researchers evaluated GCH on three widely used benchmarks: MIRFLICKR-25K, NUS-WIDE, and Microsoft COCO (MS-COCO). GCH consistently outperformed state-of-the-art (SOTA) methods across all datasets and hash bit lengths.

Performance comparison (mAP) of GCH against state-of-the-art methods on three benchmark datasets.

Key findings from the experiments include:

  • Significant Accuracy Gains: On the MIRFLICKR-25K dataset, GCH achieved a Mean Average Precision (mAP) of approximately 0.83 to 0.91, representing a massive jump over previous deep learning methods like DCMH.
  • Robustness on Complex Data: The performance gap was even more noticeable on complex datasets like MS-COCO, where GCH improved retrieval accuracy by an average of 17-18% compared to the next best deep hashing competitors.
  • Ablation Success: When the GCN was replaced with standard fully-connected layers, performance dropped sharply, proving that graph-based structural learning is the critical component of the model.

Conclusion

GCH demonstrates that incorporating structural relationships through Graph Convolutional Networks significantly improves cross-modal retrieval accuracy. While the model shows impressive results, future work will address scalability challenges for billion-item datasets and explore unsupervised scenarios where ground-truth labels are unavailable for graph construction.

Summary updated

Abstract

Deep network based cross-modal retrieval has recently made significant progress. However, bridging modality gap to further enhance the retrieval accuracy still remains a crucial bottleneck. In this paper, we propose a Graph Convolutional Hashing (GCH) approach, which learns modality-unified binary codes via an affinity graph. An end-to-end deep architecture is constructed with three main components: a semantic encoder module, two feature encoding networks, and a graph convolutional network (GCN). We design a semantic encoder as a teacher module to guide the feature encoding process, a.k.a. student module, for semantic information exploiting. Furthermore, GCN is utilized to explore the inherent similarity structure among data points, which will help to generate discriminative hash codes. Extensive experiments on three benchmark datasets demonstrate that the proposed GCH outperforms the state-of-the-art methods.
BibTeX
@inproceedings{xu2019graph,
  author = {Ruiqing Xu and Chao Li and Junchi Yan and Cheng Deng and Xianglong Liu},
  title = {{Graph Convolutional Network Hashing for Cross-Modal Retrieval}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=zwuwfG3pJCo},
}
Figure