Composite Correlation Quantization for Efficient Multimodal Retrieval

Mingsheng Long, Yue Cao, Jianmin Wang, Philip S. Yu

Abstract

Efficient similarity retrieval from large-scale multimodal databases is pervasive in modern search engines and social networks. To support queries across content modalities, systems must enable cross-modal correlation and computation-efficient indexing. While hashing methods have shown great potential, current attempts generally fail to learn isomorphic hash codes in a seamless scheme. Typically, they embed multiple modalities in a continuous isomorphic space and separately threshold embeddings into binary codes, which incurs substantial loss of retrieval accuracy. In this paper, we approach seamless multimodal hashing by proposing a novel Composite Correlation Quantization (CCQ) model. Specifically, CCQ jointly finds correlation-maximal mappings that transform different modalities into an isomorphic latent space and learns composite quantizers that convert these features into compact binary codes. An optimization framework is devised to preserve both intra-modal similarity and inter-modal correlation through minimizing both reconstruction and quantization errors. A comprehensive set of experiments clearly show the superior effectiveness and efficiency of CCQ against state-of-the-art hashing methods for both unimodal and cross-modal retrieval.

Introduction

In modern search engines, multimodal retrieval—such as using an image to find relevant tags or a text query to find related images—is a critical capability. To perform this at scale, researchers use hashing to convert high-dimensional data into short binary strings, allowing for lightning-fast searches using Hamming distances.

However, existing multimodal hashing methods suffer from a "two-step" pipeline: first mapping modalities into a shared continuous space, and then binarizing those values. This second step is often "blind" to the first, introducing massive quantization error and losing the fine-grained semantic correlations established during embedding. This paper introduces Composite Correlation Quantization (CCQ) to bridge this gap by treating embedding and quantization as a single, unified optimization problem.

The CCQ Framework

CCQ moves away from the disjointed two-step process in favor of a "seamless" hashing framework. It jointly optimizes for two primary goals: finding correlation-maximal mappings (inspired by Canonical Correlation Analysis) and learning composite quantizers.

Comparison between traditional two-step hashing and the proposed seamless CCQ framework.

Instead of simple thresholding, CCQ represents each latent vector as a sum of codewords selected from multiple codebooks. This composite quantization is far more expressive than standard binary hashing. The model minimizes a loss function that accounts for both reconstruction error (representation quality) and quantization error (information loss during binarization) simultaneously. Furthermore, CCQ can leverage "partially paired" data, using large amounts of unimodal data to better define underlying distributions when full image-text pairs are unavailable.

Experimental Evaluation

The authors evaluated CCQ on three standard benchmarks: NUS-WIDE (~195,000 pairs), Wiki (~2,800 pairs), and Flickr1M (1 million images for scalability testing).

CCQ consistently outperformed state-of-the-art unsupervised methods (like CVH and IMH) and even surpassed supervised methods (like SCM and QCH) in many tasks. A key finding is that while other methods see a massive drop in Mean Average Precision (MAP) when moving from continuous features to binary codes, CCQ remains "almost lossless."

Precision vs. number of retrieved items (R) for Image-to-Text retrieval at 32 bits, showing CCQ's superior performance.

On the Flickr1M dataset, CCQ demonstrated linear growth in training time and constant memory usage via mini-batching, proving its readiness for industrial-scale databases. Interestingly, the model often performed better at Image → Text retrieval than Image → Image, suggesting that the joint learning successfully transfers rich semantic information from the text modality to the visual one.

Conclusion and Future Work

CCQ represents a significant advancement in multimodal retrieval by integrating the binary constraint directly into the correlation learning loop. This ensures the resulting latent space is inherently "quantization-friendly." While the current model relies on linear transformations to map data into the latent space, future work could integrate Deep Neural Networks (such as CNNs) to learn more abstract, non-linear features before the quantization step, potentially further increasing retrieval accuracy in complex visual domains.

Summary updated

Abstract

Efficient similarity retrieval from large-scale multimodal database is pervasive in modern search engines and social networks. To support queries across content modalities, the system should enable cross-modal correlation and computation-efficient indexing. While hashing methods have shown great potential in achieving this goal, current attempts generally fail to learn isomorphic hash codes in a seamless scheme, that is, they embed multiple modalities in a continuous isomorphic space and separately threshold embeddings into binary codes, which incurs substantial loss of retrieval accuracy. In this paper, we approach seamless multimodal hashing by proposing a novel Composite Correlation Quantization (CCQ) model. Specifically, CCQ jointly finds correlation-maximal mappings that transform different modalities into isomorphic latent space, and learns composite quantizers that convert the isomorphic latent features into compact binary codes. An optimization framework is devised to preserve both intra-modal similarity and inter-modal correlation through minimizing both reconstruction and quantization errors, which can be trained from both paired and partially paired data in linear time. A comprehensive set of experiments clearly show the superior effectiveness and efficiency of CCQ against the state of the art hashing methods for both unimodal and cross-modal retrieval.
BibTeX
@inproceedings{long2016composite,
  author = {Mingsheng Long and Yue Cao and Jianmin Wang and Philip S. Yu},
  title = {{Composite Correlation Quantization for Efficient Multimodal Retrieval}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2016},
  url = {https://openreview.net/forum?id=HkN87B-_Zr},
}
Figure