Simultaneous Feature Aggregating and Hashing for Compact Binary Code Learning

Thanh-Toan Do, Khoa Le, Tuan Hoang, Huu Le, Tam V. Nguyen, Ngai-Man Cheung

TL;DR

This paper proposes a novel framework that jointly optimizes image feature aggregation and binary hashing, rather than treating them as independent sequential steps. By making the aggregation process "aware" of the subsequent hashing constraints, the authors produce more discriminative binary codes that are significantly faster to train and more accurate for large-scale image retrieval.

What Problem Does This Paper Address?

In modern image retrieval, the standard pipeline follows a strict sequence: extract local features (like SIFT), embed them into a high-dimensional space, aggregate them into a single global vector (e.g., VLAD or Fisher Vectors), and finally apply a hashing function to compress that vector into a compact binary code.

The authors identify a major flaw: aggregation and hashing are usually designed independently. An aggregation method might produce a global vector that is excellent for Euclidean distance but difficult to reconstruct using binary bits. This "semantic gap" between the two stages leads to suboptimal retrieval performance. Furthermore, existing state-of-the-art hashing methods like the Binary Autoencoder (BA) are computationally expensive because they involve solving NP-hard optimization problems.

Methods

The authors introduce three key components to solve these issues:

  1. Relaxed Binary Autoencoder (RBA): To speed up training, they propose a relaxed version of the standard Binary Autoencoder. Instead of dealing with the non-differentiable sign function and hard binary constraints directly, they use a quadratic penalty method with an auxiliary variable. This allows for an alternating optimization strategy with closed-form solutions, making it over 10 times faster than the original BA while maintaining or improving accuracy.
  2. Simultaneous Feature Aggregating and Hashing (SAH): This is the unsupervised framework. It jointly optimizes the hashing parameters and the aggregated representations. The objective function ensures the resulting global vectors satisfy two properties: they must be close to the original local features (aggregation property) and they must be easily reconstructible by binary codes (hashing property).
  3. Supervised SAH (SASH): When labels are available, the framework is extended to ensure the binary codes preserve semantic similarity. Since test images lack labels, the authors include a clever "mapping" step ($P$) that learns to project standard aggregated features into the optimized "learned" feature space.

Datasets

The researchers evaluated their methods across a wide variety of standard benchmarks:

  • Unsupervised Hashing: CIFAR10, MNIST, SIFT1M, Holidays, and Oxford5k.
  • Supervised Hashing: CIFAR10, MNIST, and NUS-WIDE.
  • Features: They tested both hand-crafted features (SIFT) and deep features (activations from VGG-16 convolutional and fully-connected layers).

Key Insights and Ideas

  • Joint Optimization is Key: The "aha moment" is that the aggregation step shouldn't just summarize local features; it should summarize them in a way that hashing can easily handle. By coupling these steps, the resulting binary codes are more "reconstructible" and thus more discriminative.
  • Efficiency through Relaxation: By transforming an NP-hard binary constraint into a relaxed optimization problem with an auxiliary variable, the authors proved that you don't need to sacrifice speed for accuracy in autoencoder-based hashing.
  • Bridging the Supervised Gap: In the supervised setting (SASH), the authors recognized that the optimized features ($\Phi$) rely on labels that aren't available at test time. Their solution—learning a linear mapping $P$ from the "easy to compute" features to the "optimized" features—effectively acts as a simplified distance metric learning step.

Key Results

  • Speed: RBA is more than 10x faster to train than the standard Binary Autoencoder (BA) across all tested bit lengths.
  • Unsupervised Performance: On the Oxford5k dataset, SAH outperformed the best competitor (RBA/BA) by roughly 2-3% mAP (mean Average Precision).
  • Supervised Performance: SASH showed dramatic improvements over traditional supervised methods. On CIFAR10, SASH achieved a mAP of ~53-63%, whereas the popular SDH method hovered around 40-48%.
  • Deep Learning Comparison: SASH proved competitive with, and often superior to, end-to-end deep supervised hashing methods (like DQN and DHN), particularly in "unseen class" retrieval scenarios where the model must generalize to categories it didn't see during training.

Limitations and Future Work

While the paper demonstrates significant gains, a few areas remain for exploration:

  • Linear Mapping Limitation: The mapping $P$ used in SASH is a linear ridge regression. While effective, a non-linear mapping (like a small MLP) might capture more complex relationships between original and optimized features.
  • Pre-trained Dependency: The current SAH/SASH framework uses pre-trained CNN features as "local" inputs. While this is efficient, a fully end-to-end version that includes the convolutional layers in the joint optimization could potentially yield even higher performance, albeit at a much higher computational cost.
  • Hyper-parameter Sensitivity: The framework introduces several penalty parameters ($\lambda, \beta, \gamma, \mu$). While the authors provide cross-validation ranges, the performance is somewhat dependent on tuning these for specific datasets.

Summary updated

Abstract

Representing images by compact hash codes is an attractive approach for large-scale content-based image retrieval. In most state-of-the-art hashing-based image retrieval systems, for each image, local descriptors are first aggregated as a global representation vector. This global vector is then subjected to a hashing function to generate a binary hash code. In previous works, the aggregating and the hashing processes are designed independently. Hence, these frameworks may generate suboptimal hash codes. In this paper, we first propose a novel unsupervised hashing framework in which feature aggregating and hashing are designed simultaneously and optimized jointly. Specifically, our joint optimization generates aggregated representations that can be better reconstructed by some binary codes. This leads to more discriminative binary hash codes and improved retrieval accuracy. In addition, the proposed method is flexible. It can be extended for supervised hashing. When the data label is available, the framework can be adapted to learn binary codes which minimize the reconstruction loss with respect to label vectors. Furthermore, we also propose a fast version of the state-of-the-art hashing method Binary Autoencoder to be used in our proposed frameworks. Extensive experiments on benchmark datasets under various settings show that the proposed methods outperform the state-of-the-art unsupervised and supervised hashing methods.
BibTeX
@article{do2019simultaneous,
  author = {Thanh-Toan Do and Khoa Le and Tuan Hoang and Huu Le and Tam V. Nguyen and Ngai-Man Cheung},
  title = {{Simultaneous Feature Aggregating and Hashing for Compact Binary Code Learning}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=QGFZgVhtBkw},
}

Loading related content...

Figure