Abstract
In this paper, we propose a novel deep generative approach to cross-modal retrieval to learn hash functions in the absence of paired training samples through the cycle consistency loss. Our proposed approach employs an adversarial training scheme to learn a couple of hash functions enabling translation between modalities while assuming the underlying semantic relationship. To induce the hash codes with semantics to the input-output pair, cycle consistency loss is further proposed upon the adversarial training to strengthen the correlations between inputs and corresponding outputs. Our approach is generative to learn hash functions such that the learned hash codes can maximally correlate each input-output correspondence, meanwhile can also regenerate the inputs so as to minimize the information loss. The learning to hash embedding is thus performed to jointly optimize the parameters of the hash functions across modalities as well as the associated generative models. Extensive experiments on a variety of large-scale cross-modal data sets demonstrate that our proposed method achieves better retrieval results than the state-of-the-arts.
1. Introduction
Cross-modal retrieval—the ability to find relevant images given a text query or vice versa—is a cornerstone of modern multimedia search. The standard approach involves mapping different modalities into a shared "Hamming space" where similar items have similar binary hash codes. However, traditional methods face two major hurdles: the supervision gap, where large datasets of perfectly aligned pairs are often unavailable, and significant information loss during the compression of high-dimensional data into compact binary bits.
To address these challenges, we introduce Cycle-Consistent Deep Generative Hashing (CYC-DGH). This framework enables cross-modal retrieval without requiring paired training data by combining cycle-consistency losses with adversarial training. By treating hashing as a generative process, the model ensures that the resulting compact binary hash codes can reconstruct the original inputs, preserving crucial semantic details.
2. Methodology
CYC-DGH treats hashing as a generative process rather than a simple projection. The architecture is built upon three core components:
- Cycle-Consistent Adversarial Learning: Inspired by CycleGAN, the model trains two mappings ($G$ and $F$) to translate between image and text domains. By enforcing that a translation from image to text and back to image should return to the original input, the model discovers underlying semantic relationships even without explicit pairs.
- Deep Generative Hashing (DGH): The hash code acts as a latent representation in a generative model. By minimizing reconstruction error, the model ensures the hash codes preserve maximum information from the original domain.
- Stochastic Neurons: To handle the non-differentiable nature of discrete binary codes during training, the framework utilizes stochastic neurons and distributional derivatives, allowing gradient flow through the discrete hashing layer.
By framing hashing as a generative task, we demonstrate that a high-quality hash code must contain sufficient information to rebuild the original data. This shift from discriminative to generative modeling helps bridge the "semantic gap" inherent in cross-modal tasks.
3. Experimental Setup
The model was evaluated on three benchmark datasets:
- Microsoft COCO: Over 80,000 training images, each with five sentence annotations across 80 categories.
- IAPR TC-12: 20,000 images with associated text captions from diverse domains.
- Wiki: 2,866 Wikipedia documents consisting of an image and at least 70 words of text.
4. Key Results
CYC-DGH consistently outperformed state-of-the-art (SOTA) methods across all datasets and bit lengths (16, 32, 64, and 128 bits). On the Microsoft COCO dataset (Image $\to$ Text task at 32 bits), CYC-DGH achieved a Mean Average Precision (mAP) of 0.754, significantly higher than previous methods like SePH (0.613) and CMDVH (0.697).
The model also demonstrated superior information preservation, showing much lower $L_2$ reconstruction errors compared to standard methods like Iterative Quantization (ITQ). Ablation studies confirmed that both the GAN loss and the cycle-consistency loss are essential for effective unpaired cross-modal learning.
5. Conclusion
CYC-DGH provides a robust framework for cross-modal retrieval in unsupervised or unpaired settings. While the model assumes an underlying shared semantic distribution between modalities, it successfully bridges the semantic gap through generative modeling. Future work will explore more advanced generative architectures, such as Variational Autoencoders, to further improve retrieval quality and scalability for web-scale multimedia datasets.
Summary updated