Ranking-Based Deep Cross-Modal Hashing

Xuanwu Liu, Guoxian Yu, Carlotta Domeniconi, Jun Wang, Yazhou Ren, Maozu Guo

Introduction

In the era of big data, cross-modal retrieval—such as searching for images using text queries—has become a critical necessity. To achieve the required speed and efficiency, Ranking-Based Deep Cross-Modal Hashing (RDCMH) compresses high-dimensional data into short binary codes, enabling rapid searches in Hamming space.

Unlike traditional methods that rely on simple binary similarity labels, RDCMH introduces a semi-supervised ranking metric. This approach captures nuanced, multi-level relationships between objects, moving beyond "similar or not" to understand that an image of a "dog in a park" is more closely related to a "dog on a beach" than to a "cat in a park."

The RDCMH Framework

RDCMH addresses three primary flaws in existing hashing methods: feature mismatch from hand-crafted descriptors, oversimplified pairwise similarity, and a heavy reliance on massive labeled datasets. To solve these, the authors proposed a unified deep learning architecture.

The RDCMH framework architecture, showing the parallel image and text networks feeding into a unified loss function.

1. Dual-Network Architecture

The framework utilizes two distinct deep neural networks to learn features and hash codes simultaneously:

  • Image Modality: A Convolutional Neural Network (CNN-F) extracts visual features directly from raw pixels.
  • Text Modality: A two-layer Multi-Layer Perceptron (MLP) processes Bag-of-Words (BOW) representations.

2. Semi-supervised Semantic Ranking

RDCMH leverages both labeled and unlabeled data. For labeled data, it calculates similarity based on shared labels. For unlabeled data, it utilizes the cosine similarity of raw features. This hybrid metric constructs a "ranking list" for every query, allowing the model to exploit the structural information of unlabeled datasets.

3. Weighted Triplet Ranking Loss

The networks are optimized using a Weighted Triplet Ranking Loss. This loss function assigns higher importance to triplets with larger semantic differences, ensuring that the resulting hash codes preserve a multi-level ranking order. The system is refined using the Alternating Direction Method of Multipliers (ADMM) to balance network weight updates and binary code generation.

Experimental Evaluation

The model was evaluated on three benchmark datasets: NUS-WIDE (~260,000 web images), Mirflickr (25,000 Flickr images), and Wiki (2,866 Wikipedia documents). RDCMH consistently outperformed seven state-of-the-art baselines, including DCMH and CDQ.

Key Results and Ablation Study

RDCMH achieved significant performance gains, particularly on the Mirflickr dataset with a Mean Average Precision (MAP) of approximately 0.77–0.80. A critical finding was the importance of joint optimization; ablation studies showed that learning features and hashing independently (isolated learning) significantly degraded performance.

Performance comparison (MAP) of RDCMH and its variants across different code lengths on the Mirflickr dataset.

As shown in the ablation study above, the full RDCMH model outperforms variants that remove specific components like joint learning (NJ) or semantic ranking (NS). Furthermore, the model proved robust to weak supervision. In tests where 70% of training labels were removed, RDCMH maintained superior performance compared to deep learning competitors by utilizing the underlying data distribution to fill semantic gaps.

Conclusion and Future Work

RDCMH demonstrates that ranking-based metrics provide a richer semantic foundation for cross-modal hashing than pairwise labels. While highly effective, future work could improve the framework by incorporating modern Transformer-based text embeddings (like BERT) and optimizing the triplet sampling process to scale to even larger datasets. Currently, the model provides a robust, state-of-the-art solution for efficient and nuanced cross-modal retrieval.

Summary updated

Abstract

Cross-modal hashing has been receiving increasing interests for its low storage cost and fast query speed in multi-modal data retrievals. However, most existing hashing methods are based on hand-crafted or raw level features of objects, which may not be optimally compatible with the coding process. Besides, these hashing methods are mainly designed to handle simple pairwise similarity. The complex multilevel ranking semantic structure of instances associated with multiple labels has not been well explored yet. In this paper, we propose a ranking-based deep cross-modal hashing approach (RDCMH). RDCMH firstly uses the feature and label information of data to derive a semi-supervised semantic ranking list. Next, to expand the semantic representation power of hand-crafted features, RDCMH integrates the semantic ranking information into deep cross-modal hashing and jointly optimizes the compatible parameters of deep feature representations and of hashing functions. Experiments on real multi-modal datasets show that RDCMH outperforms other competitive baselines and achieves the state-of-the-art performance in cross-modal retrieval applications.
BibTeX
@inproceedings{liu2019ranking,
  author = {Xuanwu Liu and Guoxian Yu and Carlotta Domeniconi and Jun Wang and Yazhou Ren and Maozu Guo},
  title = {{Ranking-Based Deep Cross-Modal Hashing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33014400},
}
Figure