Abstract
Cross-modal hashing has been receiving increasing interests for its low storage cost and fast query speed in multi-modal data retrievals. However, most existing hashing methods are based on hand-crafted or raw level features of objects, which may not be optimally compatible with the coding process. Besides, these hashing methods are mainly designed to handle simple pairwise similarity. The complex multilevel ranking semantic structure of instances associated with multiple labels has not been well explored yet. In this paper, we propose a ranking-based deep cross-modal hashing approach (RDCMH). RDCMH firstly uses the feature and label information of data to derive a semi-supervised semantic ranking list. Next, to expand the semantic representation power of hand-crafted features, RDCMH integrates the semantic ranking information into deep cross-modal hashing and jointly optimizes the compatible parameters of deep feature representations and of hashing functions. Experiments on real multi-modal datasets show that RDCMH outperforms other competitive baselines and achieves the state-of-the-art performance in cross-modal retrieval applications.
Introduction
In the era of big data, cross-modal retrieval—the ability to search for one data type (e.g., images) using another (e.g., text)—has become indispensable. Hashing is a preferred solution due to its efficiency in compressing high-dimensional data into short binary codes, facilitating rapid search and minimal storage. However, traditional methods often struggle with feature incompatibility, oversimplified pairwise similarity metrics, and a heavy reliance on large labeled datasets.
Methodology
The proposed RDCMH (Ranking-based Deep Cross-modal Hashing) framework addresses these challenges through a unified, end-to-end architecture that jointly optimizes feature learning and hash code generation.
1. Semi-supervised Semantic Ranking
RDCMH constructs a semantic ranking list for each data point. It prioritizes label similarity when available and utilizes feature-level similarity for unlabeled data. This semi-supervised approach allows the model to capture fine-grained semantic structures even in "weakly-supervised" scenarios.
2. Deep Feature Learning
The framework employs two distinct networks:
- Image Modality: An eight-layer CNN (CNN-F) that learns visual features directly from raw pixels.
- Text Modality: A two-layer fully connected network processing Bag-of-Words (BoW) vectors.
3. Joint Optimization
Unlike standard pipelines that separate feature extraction from hashing, RDCMH optimizes both simultaneously. It utilizes a triplet ranking loss to preserve semantic order and a quantitative loss to ensure continuous network outputs are effectively binarized. Optimization is performed using the Alternating Direction Method of Multipliers (ADMM).
Experimental Evaluation
RDCMH was evaluated on three major benchmarks: MIRFLICKR-25K, NUS-WIDE, and Wiki. The model consistently achieved state-of-the-art performance, significantly outperforming baselines like DCMH and SePH.
Key results include:
- Superior Accuracy: On MIRFLICKR (Image vs. Text, 16 bits), RDCMH achieved a Mean Average Precision (MAP) of 0.7723, surpassing the previous state-of-the-art (DCMH) at 0.7411.
- Ablation Study: As shown in the figure above, the full RDCMH model outperforms its variants (NW, ND, NS, NJ), proving the necessity of its integrated components like deep learning and joint optimization.
- Label Robustness: In tests where 70% of training labels were removed, RDCMH maintained high performance, demonstrating the effectiveness of its semi-supervised ranking mechanism.
Key Insights
- Ranking over Pairwise: Moving to a ranking-based similarity model captures the nuanced, multilevel relationships inherent in multi-label data.
- End-to-End Synergy: Feature learning and hashing are mutually beneficial; the CNN learns more discriminative features when guided by the specific requirements of the ranking loss.
- Adaptive Weighting: By assigning larger weights to more relevant points in the triplet loss, the model focuses on preserving the most critical semantic relationships.
Limitations and Future Work
While effective, triplet-based learning introduces computational complexity ($O(n^3)$), which the authors mitigate via sampling. Future work could explore more advanced sampling strategies and extend the framework to more than two modalities, such as audio or video. Additionally, the sensitivity to the hyperparameter $\lambda$ (balancing ranking and quantization loss) suggests that automated tuning could further enhance robustness across diverse datasets.
Summary updated