Transitive Hashing Network for Heterogeneous Multimedia Retrieval

Zhangjie Cao, Mingsheng Long, Qiang Yang

TL;DR

The Transitive Hashing Network (THN) enables efficient cross-modal multimedia retrieval (e.g., using text queries to search an image database) even when no direct relationship or paired data exists between the query and database domains during training. By leveraging an auxiliary dataset and aligning domain distributions, THN successfully transfers cross-modal knowledge to achieve state-of-the-art retrieval performance.

What Problem Does This Paper Address?

Cross-modal retrieval is essential for modern search engines and social networks, allowing users to find relevant data across different modalities, such as retrieving images using text queries. To make this process fast and storage-efficient at scale, researchers use cross-modal hashing, which compresses high-dimensional data into compact binary codes.

However, traditional cross-modal hashing methods suffer from a severe limitation: they assume that direct, heterogeneous relationships (like paired images and text captions) are available within the target query/database domain during training. In many practical scenarios, this assumption fails. For instance, a user might want to search for images on ImageNet using text questions from YahooQA. Because these two datasets are completely unlinked, traditional methods struggle to bridge the gap.

Figure 1: Problem overview. (left) Traditional cross-modal hashing, where heterogeneous relationship between query and database (black arrows) is available for hash learning. (right) The new transitive hashing, where heterogeneous relationship is not directly available between query and database (dashed arrows) but is available from an auxiliary dataset of different distributions (purple arrows).

The authors address this challenge by introducing transitive hashing, which eliminates the need for direct relationship links between the target query and database domains by utilizing an independent, paired auxiliary dataset.

Methods

The authors propose the Transitive Hashing Network (THN), a hybrid deep learning architecture consisting of two main components: an image network (based on AlexNet) and a text network (a multilayer perceptron). Both networks terminate in a specialized hash layer ($fch$) that outputs compact binary codes.

Figure 2: The hybrid architecture of Transitive Hashing Network (THN), which comprises heteroge- neous relationship learning, homogeneous distribution alignment, and quantization error minimization. The key is to build a transitivity (in purple) from query to database across both modalities and domains.

To achieve transitive retrieval without target domain pairs, THN integrates two key mechanisms into a unified optimization framework:

  1. Heterogeneous Relationship Learning: THN learns the correlations between different modalities using an auxiliary dataset (e.g., Flickr data containing paired images and texts). It employs a Bayesian framework with a novel pairwise cross-entropy loss to preserve semantic similarities, alongside a pairwise quantization loss to minimize the information lost when converting continuous network activations into discrete binary hash codes.
  2. Homogeneous Distribution Alignment: Because the auxiliary dataset and the target query/database domains come from different distributions, a substantial "dataset shift" exists. THN uses Maximum Mean Discrepancy (MMD) to minimize the distribution gap between the auxiliary domain and the target domains in the Hamming space, ensuring that the learned cross-modal knowledge transfers effectively.

Datasets

The model was evaluated on two primary benchmarks:

  • NUS-WIDE: A popular dataset containing 269,648 image-text pairs. The authors pruned it to 16 shared semantic categories for evaluation.
  • ImageNet-YahooQA: A heterogeneous dataset specifically constructed to evaluate the transitive setup. It combines images from ImageNet and text questions/answers from YahooQA. Because there are no direct links between ImageNet and YahooQA, NUS-WIDE was utilized as the supervised auxiliary dataset, while unsupervised data from ImageNet and YahooQA was used to align the distributions.

Key Insights and Ideas

  • Transitivity via Distribution Alignment: The core "aha moment" is that cross-modal retrieval between unlinked datasets is entirely possible if you can successfully align their individual distributions to a shared, well-mapped auxiliary space.
  • Controlling Quantization Error: Standard continuous relaxation in deep hashing often introduces large approximation errors during final binarization. THN introduces a cross-entropy prior over continuous activations, which forces the network to output values closer to binary constraints ($\{-1, 1\}$) during the training phase, preserving retrieval quality.

Key Results

THN was compared against several state-of-the-art cross-modal hashing methods, including unsupervised methods (CVH, IMH), supervised methods (QCH, HTH), and a deep hashing baseline (DCMH).

  • NUS-WIDE (Standard Setup): THN outperformed all baselines. Compared to the deep hashing method DCMH, THN achieved relative Mean Average Precision (MAP) increases of 9.47% for Image-to-Text ($I \rightarrow T$) and 2.85% for Text-to-Image ($T \rightarrow I$) tasks.
  • ImageNet-YahooQA (Transitive Setup): In the more challenging cross-domain, cross-modal task, THN outperformed the competition by large margins. It achieved relative MAP increases of 5.03% ($I \rightarrow T$) and 6.91% ($T \rightarrow I$) over DCMH, demonstrating the power of its distribution alignment module.

Limitations and Future Work

While the paper demonstrates strong performance, a natural limitation is the reliance on a high-quality auxiliary dataset that shares semantic categories with the target domains. For future work, the authors plan to extend this transitive deep hashing approach to more complex, real-world online social media applications.

Summary updated

Abstract

Hashing has been widely applied to large-scale multimedia retrieval due to the storage and retrieval efficiency. Cross-modal hashing enables efficient retrieval from database of one modality in response to a query of another modality. Existing work on cross-modal hashing assumes heterogeneous relationship across modalities for hash function learning. In this paper, we relax the strong assumption by only requiring such heterogeneous relationship in an auxiliary dataset different from the query/database domain. We craft a hybrid deep architecture to simultaneously learn the cross-modal correlation from the auxiliary dataset, and align the dataset distributions between the auxiliary dataset and the query/database domain, which generates transitive hash codes for heterogeneous multimedia retrieval. Extensive experiments exhibit that the proposed approach yields state of the art multimedia retrieval performance on public datasets, i.e. NUS-WIDE, ImageNet-YahooQA.
BibTeX
@inproceedings{cao2017transitive,
  author = {Zhangjie Cao and Mingsheng Long and Qiang Yang},
  title = {{Transitive Hashing Network for Heterogeneous Multimedia Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.1609/aaai.v31i1.10487},
}
Figure