TL;DR
The Transitive Hashing Network (THN) enables efficient cross-modal multimedia retrieval (e.g., using text queries to search an image database) even when no direct relationship or paired data exists between the query and database domains during training. By leveraging an auxiliary dataset and aligning domain distributions, THN successfully transfers cross-modal knowledge to achieve state-of-the-art retrieval performance.
What Problem Does This Paper Address?
Cross-modal retrieval is essential for modern search engines and social networks, allowing users to find relevant data across different modalities, such as retrieving images using text queries. To make this process fast and storage-efficient at scale, researchers use cross-modal hashing, which compresses high-dimensional data into compact binary codes.
However, traditional cross-modal hashing methods suffer from a severe limitation: they assume that direct, heterogeneous relationships (like paired images and text captions) are available within the target query/database domain during training. In many practical scenarios, this assumption fails. For instance, a user might want to search for images on ImageNet using text questions from YahooQA. Because these two datasets are completely unlinked, traditional methods struggle to bridge the gap.
The authors address this challenge by introducing transitive hashing, which eliminates the need for direct relationship links between the target query and database domains by utilizing an independent, paired auxiliary dataset.
Methods
The authors propose the Transitive Hashing Network (THN), a hybrid deep learning architecture consisting of two main components: an image network (based on AlexNet) and a text network (a multilayer perceptron). Both networks terminate in a specialized hash layer ($fch$) that outputs compact binary codes.
To achieve transitive retrieval without target domain pairs, THN integrates two key mechanisms into a unified optimization framework:
- Heterogeneous Relationship Learning: THN learns the correlations between different modalities using an auxiliary dataset (e.g., Flickr data containing paired images and texts). It employs a Bayesian framework with a novel pairwise cross-entropy loss to preserve semantic similarities, alongside a pairwise quantization loss to minimize the information lost when converting continuous network activations into discrete binary hash codes.
- Homogeneous Distribution Alignment: Because the auxiliary dataset and the target query/database domains come from different distributions, a substantial "dataset shift" exists. THN uses Maximum Mean Discrepancy (MMD) to minimize the distribution gap between the auxiliary domain and the target domains in the Hamming space, ensuring that the learned cross-modal knowledge transfers effectively.
Datasets
The model was evaluated on two primary benchmarks:
- NUS-WIDE: A popular dataset containing 269,648 image-text pairs. The authors pruned it to 16 shared semantic categories for evaluation.
- ImageNet-YahooQA: A heterogeneous dataset specifically constructed to evaluate the transitive setup. It combines images from ImageNet and text questions/answers from YahooQA. Because there are no direct links between ImageNet and YahooQA, NUS-WIDE was utilized as the supervised auxiliary dataset, while unsupervised data from ImageNet and YahooQA was used to align the distributions.
Key Insights and Ideas
- Transitivity via Distribution Alignment: The core "aha moment" is that cross-modal retrieval between unlinked datasets is entirely possible if you can successfully align their individual distributions to a shared, well-mapped auxiliary space.
- Controlling Quantization Error: Standard continuous relaxation in deep hashing often introduces large approximation errors during final binarization. THN introduces a cross-entropy prior over continuous activations, which forces the network to output values closer to binary constraints ($\{-1, 1\}$) during the training phase, preserving retrieval quality.
Key Results
THN was compared against several state-of-the-art cross-modal hashing methods, including unsupervised methods (CVH, IMH), supervised methods (QCH, HTH), and a deep hashing baseline (DCMH).
- NUS-WIDE (Standard Setup): THN outperformed all baselines. Compared to the deep hashing method DCMH, THN achieved relative Mean Average Precision (MAP) increases of 9.47% for Image-to-Text ($I \rightarrow T$) and 2.85% for Text-to-Image ($T \rightarrow I$) tasks.
- ImageNet-YahooQA (Transitive Setup): In the more challenging cross-domain, cross-modal task, THN outperformed the competition by large margins. It achieved relative MAP increases of 5.03% ($I \rightarrow T$) and 6.91% ($T \rightarrow I$) over DCMH, demonstrating the power of its distribution alignment module.
Limitations and Future Work
While the paper demonstrates strong performance, a natural limitation is the reliance on a high-quality auxiliary dataset that shares semantic categories with the target domains. For future work, the authors plan to extend this transitive deep hashing approach to more complex, real-world online social media applications.
Summary updated