A Novel Semantics-Preserving Hashing for Fine-Grained Image Retrieval

Han Sun, Yejia Fan, Jiaquan Shen, Ningzhong Liu, Dong Liang, Huiyu ZhouIEEE Access 2020 · OpenReview

Abstract

With the advent of the era of big data, the storage and retrieval of data have become a research hotspot. Hashing methods that transform high-dimensional data into compact binary codes have received increasing attention. Recently, with the successful application of convolutional neural networks in computer vision, deep hashing methods utilize an end-to-end framework to learn feature representations and hash codes mutually, which achieve better retrieval performance than conventional hashing methods. However, deep hashing methods still face some challenges in image retrieval. Firstly, most existing deep hashing methods preserve similarity between original data space and hash coding space using loss functions with high time complexity, which cannot get a win-win situation in time and accuracy. Secondly, few existing deep hashing methods are designed for fine-grained image retrieval, which is necessary in practice. In this study, we propose a novel semantics-preserving hashing method which solves the above problems. We add a hash layer before the classification layer as a feature switch layer to guide the classification. At the same time, we replace the complicated loss with the simple classification loss, combining with quantization loss and bit balance loss to generate high-quality hash codes. Besides, we incorporate feature extractor designed for fine-grained image classification into our network for better representation learning. The results on three widely-used fine-grained image datasets show that our method is superior to other state-of-the-art image retrieval methods.

The Challenge of Fine-Grained Retrieval

In the era of big data, efficient image retrieval is paramount. Hashing methods, which map high-dimensional data into compact binary codes, have emerged as a leading solution due to their speed and low storage requirements. However, standard hashing techniques often struggle with fine-grained image retrieval (FGIR). While general retrieval can easily distinguish a dog from a car, FGIR requires identifying subtle differences between sub-categories, such as specific dog breeds.

Fine-grained retrieval requires distinguishing between highly similar sub-categories, such as a Basset and a Walker Hound, whereas general retrieval focuses on broader categories.

Existing deep hashing methods often rely on complex pairwise or triplet loss functions to preserve similarity, leading to high computational costs ($O(n^2)$ or $O(n^3)$). Furthermore, they frequently fail to capture the localized, informative features necessary for fine-grained tasks.

The Proposed Semantics-Preserving Framework

To address these limitations, this study introduces a novel semantics-preserving hashing method. The architecture is designed to learn discriminative features and compact hash codes simultaneously through an end-to-end framework.

The proposed architecture integrates a fine-grained feature extractor (NTS-Net) with a "feature switch" hash layer and a joint learning loss part.

1. Feature Extraction with NTS-Net

Instead of a standard backbone, the model utilizes NTS-Net, which is specifically engineered for fine-grained classification. It identifies and extracts features from "informative regions" of an image—such as the specific patterns on a bird's wing—ensuring that the hash codes are based on the most relevant visual cues.

2. The "Feature Switch" Hash Layer

A key innovation is the placement of the hash layer before the classification layer. In this configuration, the hash layer acts as a "feature switch." Each bit in the resulting binary code corresponds to the presence or absence of a specific semantic attribute. This bottleneck forces the network to compress essential information into a highly efficient format that directly supports classification.

3. Simplified Joint Learning Loss

The researchers replaced computationally expensive similarity-preserving losses with a more efficient joint loss function consisting of three parts:

  • Classification Loss: Uses standard cross-entropy to ensure the features remain discriminative for category identification.
  • Quantization Loss: Minimizes the error when converting continuous neural outputs into discrete binary values (-1 or 1).
  • Bit Balance Loss: Maximizes information entropy by ensuring each bit in the hash code is utilized equally across the dataset.

Experimental Results

The method was evaluated on three prominent fine-grained datasets: CUB200-2011 (birds), Stanford Dogs, and Stanford Cars.

  • Superior Accuracy: On the CUB200-2011 dataset, the proposed method achieved a mean Average Precision (mAP) of 0.8576 for 64-bit codes, significantly outperforming previous state-of-the-art methods like FPH (0.6233).
  • Efficiency at Low Bitrates: The model demonstrated remarkable robustness; even with only 16 bits, it maintained an mAP of 0.7919 on the bird dataset, whereas competing methods saw sharp performance declines.
  • Interpretability: Analysis of the hash bits confirmed that specific bits toggle based on physical traits (e.g., ear shape or plumage color), validating the "feature switch" concept.

Conclusion

By integrating a fine-grained feature extractor with a strategically placed hash layer and a simplified loss structure, this research provides a win-win solution for accuracy and computational efficiency in image retrieval. Future work will explore extending this framework to zero-shot retrieval scenarios, where the system must identify categories not seen during training.

Summary updated

Abstract

With the advent of the era of big data, the storage and retrieval of data have become a research hotspot. Hashing methods that transform high-dimensional data into compact binary codes have received increasing attention. Recently, with the successful application of convolutional neural networks in computer vision, deep hashing methods utilize an end-to-end framework to learn feature representations and hash codes mutually, which achieve better retrieval performance than conventional hashing methods. However, deep hashing methods still face some challenges in image retrieval. Firstly, most existing deep hashing methods preserve similarity between original data space and hash coding space using loss functions with high time complexity, which cannot get a win-win situation in time and accuracy. Secondly, few existing deep hashing methods are designed for fine-grained image retrieval, which is necessary in practice. In this study, we propose a novel semantics-preserving hashing method which solves the above problems. We add a hash layer before the classification layer as a feature switch layer to guide the classification. At the same time, we replace the complicated loss with the simple classification loss, combining with quantization loss and bit balance loss to generate high-quality hash codes. Besides, we incorporate feature extractor designed for fine-grained image classification into our network for better representation learning. The results on three widely-used fine-grained image datasets show that our method is superior to other state-of-the-art image retrieval methods.
BibTeX
@inproceedings{sun2020novel,
  author = {Han Sun and Yejia Fan and Jiaquan Shen and Ningzhong Liu and Dong Liang and Huiyu Zhou},
  title = {{A Novel Semantics-Preserving Hashing for Fine-Grained Image Retrieval}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=4CGP2bN9fh},
}
Figure