Transitive Hashing Network for Heterogeneous Multimedia Retrieval

Zhangjie Cao, Mingsheng Long, Jianmin Wang, Qiang Yang

Introduction

Cross-modal hashing is a critical technique for searching massive multimedia databases, as it compresses high-dimensional data into compact binary codes to enable rapid retrieval. However, traditional methods typically rely on a "strict assumption": the availability of a training set where different modalities (e.g., images and text) are already linked or paired. In real-world scenarios, such direct supervision is often unavailable. For instance, one might wish to use text queries from a platform like YahooQA to search for images in ImageNet, despite no pre-existing links between these specific datasets.

This paper addresses the transitive retrieval problem, where the goal is to learn hashing across modalities using an auxiliary dataset (e.g., Flickr) as a "bridge." This auxiliary data contains image-text pairs, but follows a different distribution than the target query and database domains.

The transitive retrieval problem: an auxiliary dataset bridges the gap between query and database modalities that lack direct training pairs.

Methodology: The Transitive Hashing Network (THN)

The authors propose the Transitive Hashing Network (THN), an end-to-end deep learning architecture designed to construct a transitive link between heterogeneous domains. The network consists of an image branch (based on AlexNet) and a text branch (a three-layer MLP), integrated through three core modules:

  1. Heterogeneous Relationship Learning: The model learns cross-modal correlations from the auxiliary dataset using a pairwise cross-entropy loss. This ensures that semantically similar image-text pairs result in similar hash codes.
  2. Homogeneous Distribution Alignment: To combat "dataset shift" between the auxiliary and target domains, THN incorporates a Maximum Mean Discrepancy (MMD) module. This aligns the distributions in the Hamming space, ensuring that learned correlations transfer effectively to the target data.
  3. Quantization Error Minimization: A novel cross-entropy prior is introduced to force neural network activations toward binary values (-1 or 1) during training. This minimizes the semantic gap typically created during binarization.
The THN architecture, showing the integration of image and text networks with MMD alignment and pairwise quantization/cross-entropy losses.

Experimental Evaluation

The researchers evaluated THN on two major benchmarks: NUS-WIDE (standard cross-modal retrieval) and ImageNet-YahooQA (a custom transitive benchmark). In the transitive task, NUS-WIDE served as the auxiliary bridge.

Key Results

THN consistently outperformed state-of-the-art methods, including deep models like DCMH:

  • NUS-WIDE: Achieved a relative increase in Mean Average Precision (MAP) of 9.47% for Image → Text and 2.85% for Text → Image retrieval compared to previous bests.
  • ImageNet-YahooQA: The gains were more significant, with relative MAP increases of 5.03% and 6.91%, respectively.
  • Robustness: Precision-Recall curves demonstrate that THN maintains high precision even at high recall levels, proving its effectiveness across diverse retrieval scenarios.

Conclusion and Future Work

The success of THN highlights that direct supervision is not mandatory for cross-modal retrieval if a bridge dataset is available and distribution shifts are properly managed via alignment modules like MMD. Future work may involve scaling these techniques to noisier social media data or extending the framework to handle more than two modalities simultaneously.

Summary updated

Abstract

Hashing is widely applied to large-scale multimedia retrieval due to the storage and retrieval efficiency. Cross-modal hashing enables efficient retrieval of one modality from database relevant to a query of another modality. Existing work on cross-modal hashing assumes that heterogeneous relationship across modalities is available for learning to hash. This paper relaxes this strict assumption by only requiring heterogeneous relationship in some auxiliary dataset different from the query or database domain. We design a novel hybrid deep architecture, transitive hashing network (THN), to jointly learn cross-modal correlation from the auxiliary dataset, and align the data distributions of the auxiliary dataset with that of the query or database domain, which generates compact transitive hash codes for efficient cross-modal retrieval. Comprehensive empirical evidence validates that the proposed THN approach yields state of the art retrieval performance on standard multimedia benchmarks, i.e. NUS-WIDE and ImageNet-YahooQA.
BibTeX
@inproceedings{cao2017transitive,
  author = {Zhangjie Cao and Mingsheng Long and Jianmin Wang and Qiang Yang},
  title = {{Transitive Hashing Network for Heterogeneous Multimedia Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.1609/aaai.v31i1.10487},
}
Figure