Simultaneous Feature Aggregating and Hashing for Compact Binary Code Learning

Thanh-Toan Do, Khoa Le, Tuan Hoang, Huu Le, Tam V. Nguyen, Ngai-Man Cheung

Abstract

Representing images by compact hash codes is an attractive approach for large-scale content-based image retrieval. In most state-of-the-art hashing-based image retrieval systems, local descriptors are first aggregated as a global representation vector, which is then subjected to a hashing function. Traditionally, these processes are designed independently, potentially leading to suboptimal codes. This paper proposes a novel unsupervised hashing framework where feature aggregating and hashing are optimized jointly. By generating aggregated representations that are better reconstructed by binary codes, we achieve more discriminative hash codes and improved retrieval accuracy. We also introduce a fast version of the Binary Autoencoder (BA) and extend the framework for supervised hashing. Extensive experiments demonstrate that our methods outperform existing state-of-the-art techniques.

Introduction

Large-scale image retrieval relies on compressing high-dimensional image data into compact binary hash codes. The standard pipeline involves extracting local features (e.g., SIFT), aggregating them into a global vector (e.g., VLAD), and then applying a hashing function. However, the independent design of these two stages creates a "semantic gap": an aggregation method might optimize for Euclidean distance but produce vectors that are difficult to represent accurately with binary bits. Furthermore, high-performing hashing methods like the Binary Autoencoder (BA) are often computationally prohibitive due to NP-hard optimization constraints.

Methodology

To address these challenges, we propose three primary innovations:

Relaxed Binary Autoencoder (RBA)

To overcome the computational bottleneck of the standard Binary Autoencoder, we developed the Relaxed Binary Autoencoder (RBA). Instead of directly optimizing the non-differentiable sign function, RBA employs a quadratic penalty method with an auxiliary variable. This relaxation allows for an alternating optimization strategy with closed-form solutions.

Training time comparison between RBA and BA on CIFAR10 and SIFT1M datasets.

As shown in the figure above, RBA is significantly more efficient, training over 10 times faster than the original BA while maintaining or even improving retrieval accuracy.

Simultaneous Feature Aggregating and Hashing (SAH)

The SAH framework jointly optimizes hashing parameters and aggregated representations. The objective function ensures that the resulting global vectors are both faithful to the original local features and easily reconstructible by binary codes. This coupling ensures that the aggregation step is "aware" of the subsequent hashing constraints.

Supervised SAH (SASH)

For scenarios where labels are available, SASH minimizes reconstruction loss relative to label vectors. To handle test images without labels, we learn a linear mapping $P$ that projects standard aggregated features into the optimized feature space, effectively bridging the gap between training and inference.

Experimental Results

We evaluated our methods on several benchmarks, including CIFAR10, MNIST, SIFT1M, and Oxford5k, using both hand-crafted (SIFT) and deep features (VGG-16).

Retrieval Accuracy

The joint optimization in SAH leads to superior discriminative power. On the Oxford5k dataset, SAH outperformed the best competitors by approximately 2-3% mAP.

Retrieval performance (mAP %) vs. number of bits on the CIFAR10 dataset.

In supervised settings, SASH showed even more dramatic gains. On CIFAR10, SASH achieved a mAP of 53-63%, significantly outperforming the popular Supervised Discrete Hashing (SDH) method, which achieved 40-48%. SASH also proved competitive with end-to-end deep supervised hashing methods, particularly in generalizing to unseen classes.

Conclusion

By jointly optimizing feature aggregation and hashing, we have demonstrated that binary codes can be made significantly more discriminative and efficient. The Relaxed Binary Autoencoder provides a scalable foundation for this framework, reducing training times by an order of magnitude. Future work may explore non-linear mappings for supervised inference and full end-to-end integration with convolutional layers to further enhance performance.

Summary updated

Abstract

Representing images by compact hash codes is an attractive approach for large-scale content-based image retrieval. In most state-of-the-art hashing-based image retrieval systems, for each image, local descriptors are first aggregated as a global representation vector. This global vector is then subjected to a hashing function to generate a binary hash code. In previous works, the aggregating and the hashing processes are designed independently. Hence these frameworks may generate suboptimal hash codes. In this paper, we first propose a novel unsupervised hashing framework in which feature aggregating and hashing are designed simultaneously and optimized jointly. Specifically, our joint optimization generates aggregated representations that can be better reconstructed by some binary codes. This leads to more discriminative binary hash codes and improved retrieval accuracy. In addition, the proposed method is flexible. It can be extended for supervised hashing. When the data label is available, the framework can be adapted to learn binary codes which minimize the reconstruction loss w.r.t. label vectors. Furthermore, we also propose a fast version of the state-of-the-art hashing method Binary Autoencoder to be used in our proposed frameworks. Extensive experiments on benchmark datasets under various settings show that the proposed methods outperform state-of-the-art unsupervised and supervised hashing methods.
BibTeX
@article{do2019simultaneous,
  author = {Thanh-Toan Do and Khoa Le and Tuan Hoang and Huu Le and Tam V. Nguyen and Ngai-Man Cheung},
  title = {{Simultaneous Feature Aggregating and Hashing for Compact Binary Code Learning}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=QGFZgVhtBkw},
}
Figure