Multimodal Learning of Geometry-Preserving Binary Codes for Semantic Image Retrieval

Go Irie, Hiroyuki Arai, Yukinobu TaniguchiIEICE Trans. Information and Systems 2017 · DOI · OpenReview

Efficiently retrieving semantically similar images from massive databases is a cornerstone of modern computer vision. While high-dimensional feature vectors provide rich representations, their computational cost in large-scale searches is prohibitive. Binary hashing addresses this by compressing features into compact binary strings, enabling rapid searches in Hamming space. However, traditional hashing methods often fail to capture complex semantic relationships, either by over-relying on Euclidean distances or by ignoring the multimodal context (e.g., tags and captions) that often accompanies images.

The mGLP Framework

This paper introduces mGLP (Multimodal learning of Geometry-preserving Linear Projections), an unsupervised hashing method designed to overcome these limitations. The framework optimizes a joint objective function balancing three critical components:

  1. Geometry Preservation: Utilizing unsupervised sparse coding, mGLP identifies the local manifold structure of the data. It ensures that the binary codes in Hamming space maintain the reconstruction relationships found in the original feature space, effectively navigating the "neighborhood trap" where semantically similar items might be Euclidean-distant.
  2. Multimodal Correlation: By leveraging image-text pairs during training, the model aligns the binary codes of images with those of their corresponding textual descriptions. This "pulls" the visual representations toward a more semantic space.
  3. Quantization Refinement: An Orthogonal Procrustes step is employed to minimize the discrepancy between continuous projections and discrete binary bits, reducing performance loss during quantization.

Experimental Results

The authors evaluated mGLP on the a-Pascal and Microsoft COCO datasets. On a-Pascal, mGLP demonstrated a remarkable 40.7% precision improvement over the state-of-the-art CCA-ITQ for 64-bit codes. Even on the larger COCO dataset using deep CNN features, mGLP provided a consistent 5.9% performance boost.

Performance comparison of mGLP against state-of-the-art hashing methods across different bit lengths and retrieval counts.

As shown in the figure above, mGLP (red line with squares) consistently outperforms competing methods like ITQ, SH, and CVH across various bit lengths and numbers of retrieved items. The steady increase in precision with bit length (Figure 1a) highlights the method's ability to effectively utilize additional capacity.

Ablation Study and Insights

A key contribution of the paper is demonstrating the necessity of both multimodal learning and manifold preservation. The researchers conducted an ablation study comparing the full mGLP model against variants that omit specific components, such as removing manifold learning or multimodal alignment.

Ablation study on a-Pascal and COCO datasets showing the impact of multimodal and geometry-preserving components on precision.

The results in the figure above confirm that the full mGLP model achieves the highest precision. The gap between mGLP and its variants (like mGLP-P, which lacks the multimodal component) underscores the "semantic anchor" role played by textual data during the training phase.

Conclusion

mGLP offers a powerful solution for semantic image retrieval by integrating manifold geometry and multimodal context into a unified linear projection framework. While the training phase is more computationally intensive than simpler methods, the resulting encoding process is extremely efficient, making it suitable for real-time applications in large-scale databases. Future work may explore non-linear deep hashing extensions to further enhance retrieval accuracy.

Summary updated

Abstract

This paper presents an unsupervised approach to feature binary coding for efficient semantic image retrieval. Although the majority of the existing me …
BibTeX
@article{irie2017multimodal,
  author = {Go Irie and Hiroyuki Arai and Yukinobu Taniguchi},
  title = {{Multimodal Learning of Geometry-Preserving Binary Codes for Semantic Image Retrieval}},
  journal = {IEICE Trans. Information and Systems},
  year = {2017},
  doi = {10.1587/transinf.2016awi0003},
}
Figure