Deep Hashing with Semantic Hash Centers for Image Retrieval

Li Chen, Rui Liu, Yuxiang Zhou, Xudong Ma, Yong Chen, Dell Zhang

Abstract

Deep hashing has emerged as a cornerstone for large-scale image retrieval, with recent point-wise methods like Central Similarity Quantization (CSQ) and Multi-class Discriminative Scaling (MDS) setting new benchmarks. These methods typically pre-assign hash centers to classes to improve discriminability. However, existing approaches rely on data-independent algorithms to generate these centers, failing to account for the semantic relationships between categories. This paper presents the Semantic Hash Centers (SHC) framework, which generates hash centers that reflect semantic similarities—ensuring that related classes (e.g., different breeds of dogs) are closer in Hamming space than unrelated ones. Our three-stage approach involves data-dependent similarity calculation, optimization of centers under distance constraints, and deep network training. Experiments across five benchmarks demonstrate that SHC significantly outperforms state-of-the-art methods, achieving up to an 11.71% improvement in Mean Average Precision (MAP).

Introduction

The explosion of visual data necessitates efficient retrieval systems. Deep hashing addresses this by mapping high-dimensional image features into compact binary codes, enabling rapid similarity searches via hardware-level XOR operations. While point-wise hashing methods have simplified training by using fixed hash centers, their reliance on mathematical distributions (like Bernoulli) or matrices (like Hadamard) ignores the inherent hierarchy of human concepts. In these traditional systems, a "Golden Retriever" is mathematically as distant from a "Labrador" as it is from a "Toaster." This lack of semantic awareness limits the discriminative power of the resulting hash codes, particularly in fine-grained retrieval scenarios.

The SHC Framework

To bridge the gap between semantic concepts and binary representations, we propose the SHC framework, structured into three distinct phases.

1. Data-Dependent Similarity Modeling

Rather than assuming all classes are equally distinct, SHC first captures the semantic "closeness" of categories. We utilize a pre-trained ResNet34 classification network to analyze the probability distributions across classes. By observing the "confusion" between categories—where the network might assign a high probability to a "Labrador" when shown a "Golden Retriever"—we derive a similarity matrix that reflects real-world data distributions. This approach is more adaptive than static text-based embeddings, as it relies on visual features relevant to the specific dataset.

2. Optimizing Semantic Hash Centers

The core of SHC is the generation of hash centers that preserve these semantic relationships while maintaining sufficient separation. We formulate an optimization problem with two primary objectives:

  • Semantic Preservation: The Hamming distance between any two hash centers should be inversely proportional to their semantic similarity.
  • Minimum Distance Constraint: To prevent "crowding" and ensure distinctiveness, we enforce a minimum distance between centers based on the Gilbert-Varshamov bound.

Solving this NP-hard problem is achieved through the Augmented Lagrangian Method (ALM), which decomposes the global objective into manageable sub-tasks, resulting in a set of optimized binary centers that mirror the semantic geometry of the data.

3. Deep Hashing Network Training

With the semantic centers established, we train a deep hashing network to map input images to these targets. The training utilizes a dual-loss function: a central similarity loss to pull image features toward their respective semantic centers, and a quantization loss to minimize the error when converting continuous features into binary bits.

Experimental Evaluation

We evaluated SHC on five versions of three major datasets: CIFAR-100, Stanford Cars (Versions A & B), and NABirds (Versions A & B). The latter is particularly challenging due to its fine-grained nature, containing 555 classes of birds where semantic relationships are highly complex.

SHC consistently outperformed nine state-of-the-art baselines, including CSQ, MDS, and OrthoHash, across all code lengths (16, 32, and 64 bits). The performance gains were substantial:

  • MAP@100: +7.26% average improvement.
  • MAP@1000: +7.62% average improvement.
  • MAP@ALL: +11.71% average improvement.

The most significant improvements were observed in the NABirds dataset. In fine-grained scenarios where categories are visually similar, the semantic awareness of SHC allows the model to maintain a structured Hamming space that prevents different bird species from collapsing into the same hash region.

Discussion and Insights

A key finding of this research is that the geometry of Hamming space should mirror the geometry of the semantic space. Previous assumptions that maximizing distance between all centers is always optimal are challenged by SHC’s success. By allowing semantically related classes to reside closer together, the model creates a more intuitive and discriminative feature space.

Furthermore, the use of the Gilbert-Varshamov bound proves essential in low-dimensional settings. When the number of classes is high (e.g., 555 in NABirds), the available Hamming space becomes crowded. SHC’s optimization ensures that even while preserving similarity, the centers remain distinct enough to avoid misclassification.

Conclusion

The SHC framework demonstrates that incorporating semantic relationships into the generation of hash centers significantly enhances image retrieval performance. By moving away from data-independent center assignment and toward a data-dependent, optimized approach, SHC provides a robust solution for large-scale, fine-grained retrieval tasks. Future work will explore dynamic center evolution during training and the extension of semantic centers to unsupervised learning environments.

Summary updated

Abstract

Deep hashing is an effective approach for large-scale image retrieval. Current methods are typically classified by their supervision types: point-wise, pair-wise, and list-wise. Recent point-wise techniques (e.g., CSQ, MDS) have improved retrieval performance by pre-assigning a hash center to each class, enhancing the discriminability of hash codes across various datasets. However, these methods rely on data-independent algorithms to generate hash centers, which neglect the semantic relationships between classes and may degrade retrieval performance. This paper introduces the concept of semantic hash centers, building on the idea of traditional hash centers. We hypothesize that hash centers of semantically related classes should have closer Hamming distances, while those of unrelated classes should be more distant. To this end, we propose a three-stage framework, SHC, to generate hash codes that preserve semantic structure. First, we develop a classification network to identify semantic similarities between classes using a data-dependent similarity calculation that adapts to varying data distributions. Second, we introduce an optimization algorithm to generate semantic hash centers, preserving semantic relatedness while enforcing a minimum distance between centers to avoid excessively similar hash codes. Finally, a deep hashing network is trained using these semantic centers to convert images into binary hash codes. Experimental results on large-scale retrieval tasks across several public datasets show that SHC significantly improves retrieval performance. Specifically, SHC achieves average improvements of +7.26%, +7.62%, and +11.71% in MAP@100, MAP@1000, and MAP@ALL metrics, respectively, over state-of-the-art methods.
BibTeX
@article{chen2025deep,
  author = {Li Chen and Rui Liu and Yuxiang Zhou and Xudong Ma and Yong Chen and Dell Zhang},
  title = {{Deep Hashing with Semantic Hash Centers for Image Retrieval}},
  journal = {ACM Trans. Information Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=r7dPCOi2Xn},
}
Figure