Abstract
Efficient similarity retrieval from large-scale multimodal databases is pervasive in modern search engines and social networks. To support queries across content modalities, systems must enable cross-modal correlation and computation-efficient indexing. While hashing methods have shown great potential, current attempts generally fail to learn isomorphic hash codes in a seamless scheme. Typically, they embed multiple modalities in a continuous isomorphic space and separately threshold embeddings into binary codes, which incurs substantial loss of retrieval accuracy. In this paper, we approach seamless multimodal hashing by proposing a novel Composite Correlation Quantization (CCQ) model. Specifically, CCQ jointly finds correlation-maximal mappings that transform different modalities into an isomorphic latent space and learns composite quantizers that convert these features into compact binary codes. An optimization framework is devised to preserve both intra-modal similarity and inter-modal correlation through minimizing both reconstruction and quantization errors. A comprehensive set of experiments clearly show the superior effectiveness and efficiency of CCQ against state-of-the-art hashing methods for both unimodal and cross-modal retrieval.
Introduction
In modern search engines, multimodal retrieval—such as using an image to find relevant tags or a text query to find related images—is a critical capability. To perform this at scale, researchers use hashing to convert high-dimensional data into short binary strings, allowing for lightning-fast searches using Hamming distances.
However, existing multimodal hashing methods suffer from a "two-step" pipeline: first mapping modalities into a shared continuous space, and then binarizing those values. This second step is often "blind" to the first, introducing massive quantization error and losing the fine-grained semantic correlations established during embedding. This paper introduces Composite Correlation Quantization (CCQ) to bridge this gap by treating embedding and quantization as a single, unified optimization problem.
The CCQ Framework
CCQ moves away from the disjointed two-step process in favor of a "seamless" hashing framework. It jointly optimizes for two primary goals: finding correlation-maximal mappings (inspired by Canonical Correlation Analysis) and learning composite quantizers.
Instead of simple thresholding, CCQ represents each latent vector as a sum of codewords selected from multiple codebooks. This composite quantization is far more expressive than standard binary hashing. The model minimizes a loss function that accounts for both reconstruction error (representation quality) and quantization error (information loss during binarization) simultaneously. Furthermore, CCQ can leverage "partially paired" data, using large amounts of unimodal data to better define underlying distributions when full image-text pairs are unavailable.
Experimental Evaluation
The authors evaluated CCQ on three standard benchmarks: NUS-WIDE (~195,000 pairs), Wiki (~2,800 pairs), and Flickr1M (1 million images for scalability testing).
CCQ consistently outperformed state-of-the-art unsupervised methods (like CVH and IMH) and even surpassed supervised methods (like SCM and QCH) in many tasks. A key finding is that while other methods see a massive drop in Mean Average Precision (MAP) when moving from continuous features to binary codes, CCQ remains "almost lossless."
On the Flickr1M dataset, CCQ demonstrated linear growth in training time and constant memory usage via mini-batching, proving its readiness for industrial-scale databases. Interestingly, the model often performed better at Image → Text retrieval than Image → Image, suggesting that the joint learning successfully transfers rich semantic information from the text modality to the visual one.
Conclusion and Future Work
CCQ represents a significant advancement in multimodal retrieval by integrating the binary constraint directly into the correlation learning loop. This ensures the resulting latent space is inherently "quantization-friendly." While the current model relies on linear transformations to map data into the latent space, future work could integrate Deep Neural Networks (such as CNNs) to learn more abstract, non-linear features before the quantization step, potentially further increasing retrieval accuracy in complex visual domains.
Summary updated