Cycle-Consistent Deep Generative Hashing for Cross-Modal Retrieval

Lin Wu, Yang Wang, Ling Shao

TL;DR

This paper introduces CYC-DGH, a deep generative hashing framework that enables cross-modal retrieval (like searching for images using text) without requiring paired training data. By combining cycle-consistency losses with adversarial training, the model learns to translate between modalities while ensuring that the resulting compact binary hash codes can actually reconstruct the original inputs, significantly reducing information loss.

What Problem Does This Paper Address?

Cross-modal retrieval—finding a relevant image given a text query or vice versa—is a cornerstone of modern multimedia search. The standard approach is to map different modalities into a shared "Hamming space" where similar items have similar binary hash codes. However, two major hurdles remain:

  1. The Supervision Gap: Most high-performing deep hashing models require large datasets of perfectly aligned pairs (e.g., an image and its exact descriptive caption). In the real world, such labeled pairs are expensive or impossible to obtain.
  2. Information Loss: Traditional hashing is a "one-way street" that compresses high-dimensional data into a few bits. This often discards crucial semantic details, leading to poor retrieval quality.

Methods

The authors propose Cycle-Consistent Deep Generative Hashing (CYC-DGH), which treats hashing as a generative process rather than just a simple projection.

  • Cycle-Consistent Adversarial Learning: Borrowing a page from the CycleGAN playbook, the authors train two mappings ($G$ and $F$) to translate between image and text domains. Even without paired data, the model learns that if you translate an image to text and then back to an image, you should end up with something very close to the original. This "cycle-consistency" acts as a powerful regularizer.
  • Deep Generative Hashing (DGH): Instead of just outputting a code, the model learns to regenerate the input from the hash code. The hash code acts as a latent representation in a generative model. By minimizing the reconstruction error, the model ensures the hash codes preserve the maximum amount of information from the original domain.
  • Stochastic Neurons: To handle the non-differentiable nature of discrete binary codes (0s and 1s) during training, the authors use stochastic neurons and distributional derivatives, allowing the gradient to flow through the discrete hashing layer.

Datasets

The model was evaluated on three widely used benchmark datasets:

  • Microsoft COCO: A massive dataset containing over 80,000 training images, each with at least five sentence annotations across 80 categories.
  • IAPR TC-12: Consists of 20,000 images from diverse domains (sports, cities, etc.) with associated text captions.
  • Wiki: A smaller but challenging dataset of 2,866 Wikipedia documents, where each entry consists of an image and a corresponding text of at least 70 words.

Key Insights and Ideas

  • Learning from Sets, Not Pairs: The "aha moment" here is that you don't need to know which specific image matches which specific text to learn a good mapping. By matching the distribution of the entire set of images to the distribution of the set of texts, and enforcing cycle-consistency, the model discovers the underlying semantic relationships.
  • Hashing as Compression: By framing hashing as a generative task, the authors prove that a "good" hash code is one that contains enough information to rebuild the original data. This shift from discriminative to generative modeling helps bridge the "semantic gap."
  • Efficiency via Gaussian Reconstruction: Unlike some iterative quantization methods, CYC-DGH uses a Gaussian reconstruction approach that doesn't require rigid orthogonality constraints, making the training process more efficient.

Key Results

CYC-DGH consistently outperformed state-of-the-art (SOTA) methods across all datasets and bit lengths (16, 32, 64, and 128 bits).

  • Retrieval Accuracy: On the Microsoft COCO dataset (Image $\to$ Text task at 32 bits), CYC-DGH achieved a Mean Average Precision (mAP) of 0.754, significantly higher than previous SOTA methods like SePH (0.613) and CMDVH (0.697).
  • Information Preservation: The model showed much lower $L_2$ reconstruction errors compared to standard methods like Iterative Quantization (ITQ), proving that its generative approach keeps more data "alive" in the bits.
  • Ablation Success: The authors showed that removing either the GAN loss or the Cycle-consistency loss caused performance to tank, confirming that both components are essential for unpaired cross-modal learning.

Limitations and Future Work

While the model excels at learning from unpaired data, it still assumes that the two sets of data (images and text) share an underlying semantic distribution. If the two datasets are completely unrelated, the cycle-consistency might find "hallucinated" mappings.

For future work, the authors suggest exploring even more powerful generative models (like more advanced GAN architectures or Variational Autoencoders) to further improve the quality of the generated data and the resulting hash codes. They also aim to scale the approach to even larger, more diverse web-scale multimedia datasets.

Summary updated

Abstract

In this paper, we propose a novel deep generative approach to cross-modal retrieval to learn hash functions in the absence of paired training samples through the cycle consistency loss. Our proposed approach employs adversarial training scheme to learn a couple of hash functions enabling translation between modalities while assuming the underlying semantic relationship. To induce the hash codes with semantics to the input-output pair, cycle consistency loss is further delved into the adversarial training to strengthen the correlation between the inputs and corresponding outputs. Our approach is generative to learn hash functions, such that the learned hash codes can maximally correlate each input-output correspondence and also regenerate the inputs so as to minimize the information loss. The learning to hash embedding is thus performed to jointly optimize the parameters of the hash functions across modalities as well as the associated generative models. Extensive experiments on a variety of large-scale cross-modal data sets demonstrate that our proposed method outperforms the state of the arts.
BibTeX
@article{wu2019cycle,
  author = {Lin Wu and Yang Wang and Ling Shao},
  title = {{Cycle-Consistent Deep Generative Hashing for Cross-Modal Retrieval}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=RZbRJQ-U5E},
}

Loading related content...

Figure