Abstract
Hashing has been widely applied to large-scale multimedia retrieval due to the storage and retrieval efficiency. Cross-modal hashing enables efficient retrieval from database of one modality in response to a query of another modality. Existing work on cross-modal hashing assumes heterogeneous relationship across modalities for hash function learning. In this paper, we relax the strong assumption by only requiring such heterogeneous relationship in an auxiliary dataset different from the query/database domain. We craft a hybrid deep architecture to simultaneously learn the cross-modal correlation from the auxiliary dataset, and align the dataset distributions between the auxiliary dataset and the query/database domain, which generates transitive hash codes for heterogeneous multimedia retrieval. Extensive experiments exhibit that the proposed approach yields state of the art multimedia retrieval performance on public datasets, i.e. NUS-WIDE, ImageNet-YahooQA.
Introduction
Cross-modal retrieval—such as using an image to find relevant news articles—usually relies on "cross-modal hashing." This technique compresses high-dimensional data into compact binary codes, ensuring that similar objects across different modalities (images and text) end up with similar codes. However, traditional methods require a training dataset where the query modality and the database modality are already linked (e.g., images with their specific captions).
In the real world, we often want to bridge domains that have never met. For example, you might want to search a database of Yahoo! Questions and Answers using images from ImageNet. Because there are no direct links between these two datasets, standard hashing methods fail. This paper introduces the Transitive Hashing Network (THN) to address this "transitive" challenge: bridging two different modalities from two different domains without direct supervision.
Methods
The authors propose the Transitive Hashing Network (THN), a hybrid deep architecture that uses an auxiliary dataset as a bridge.
- Architecture: THN uses a dual-stream approach. One stream is a Convolutional Neural Network (AlexNet) for processing images, and the other is a Multilayer Perceptron (MLP) for text. Both streams terminate in a "hash layer" that produces $b$-bit binary codes.
- Heterogeneous Relationship Learning: Since the target query and database have no links, the model trains on an auxiliary dataset (like NUS-WIDE) that contains image-text pairs. It uses a pairwise cross-entropy loss to ensure that related items in the auxiliary set have similar hash codes.
- Homogeneous Distribution Alignment: To prevent the model from overfitting to the auxiliary data, the authors integrate Maximum Mean Discrepancy (MMD). This module aligns the distributions of the auxiliary data with the target query/database data, enabling knowledge transfer across domains.
- Quantization Control: Converting continuous activations into binary codes usually introduces errors. THN introduces a novel cross-entropy prior that forces activations toward binary values during training, minimizing quality loss during binarization.
Results and Discussion
The researchers evaluated THN on two main benchmarks:
- NUS-WIDE: A standard dataset containing ~270,000 image-text pairs, used to test performance in traditional cross-modal settings.
- ImageNet-YahooQA: The "transitive" task where the query set comes from ImageNet (images) and the database from YahooQA (text), with zero direct links between them. NUS-WIDE was used as the auxiliary bridge.
THN demonstrated state-of-the-art performance across the board:
- On NUS-WIDE: THN outperformed the previous best deep method (DCMH) by 9.47% in Image $\rightarrow$ Text retrieval and 2.85% in Text $\rightarrow$ Image retrieval.
- On ImageNet-YahooQA: THN showed even more dominance, beating the competition by 5.03% to 6.91% in average precision.
Conclusion
The core insight of THN is that direct links are not strictly necessary if the model can learn the concept of a relationship in one domain and successfully align the distributions of target domains to it. While THN requires some semantic overlap between the auxiliary and target datasets, it provides a powerful framework for bridging disparate data modalities. Future work could involve extending this architecture to handle evolving social media streams where data distributions shift in real-time.
Summary updated