Abstract
Deep hashing has emerged as a cornerstone for large-scale image retrieval, with recent point-wise methods like Central Similarity Quantization (CSQ) and Multi-class Discriminative Scaling (MDS) setting new benchmarks. These methods typically pre-assign hash centers to classes to improve discriminability. However, existing approaches rely on data-independent algorithms to generate these centers, failing to account for the semantic relationships between categories. This paper presents the Semantic Hash Centers (SHC) framework, which generates hash centers that reflect semantic similarities—ensuring that related classes (e.g., different breeds of dogs) are closer in Hamming space than unrelated ones. Our three-stage approach involves data-dependent similarity calculation, optimization of centers under distance constraints, and deep network training. Experiments across five benchmarks demonstrate that SHC significantly outperforms state-of-the-art methods, achieving up to an 11.71% improvement in Mean Average Precision (MAP).
Introduction
The explosion of visual data necessitates efficient retrieval systems. Deep hashing addresses this by mapping high-dimensional image features into compact binary codes, enabling rapid similarity searches via hardware-level XOR operations. While point-wise hashing methods have simplified training by using fixed hash centers, their reliance on mathematical distributions (like Bernoulli) or matrices (like Hadamard) ignores the inherent hierarchy of human concepts. In these traditional systems, a "Golden Retriever" is mathematically as distant from a "Labrador" as it is from a "Toaster." This lack of semantic awareness limits the discriminative power of the resulting hash codes, particularly in fine-grained retrieval scenarios.
The SHC Framework
To bridge the gap between semantic concepts and binary representations, we propose the SHC framework, structured into three distinct phases.
1. Data-Dependent Similarity Modeling
Rather than assuming all classes are equally distinct, SHC first captures the semantic "closeness" of categories. We utilize a pre-trained ResNet34 classification network to analyze the probability distributions across classes. By observing the "confusion" between categories—where the network might assign a high probability to a "Labrador" when shown a "Golden Retriever"—we derive a similarity matrix that reflects real-world data distributions. This approach is more adaptive than static text-based embeddings, as it relies on visual features relevant to the specific dataset.
2. Optimizing Semantic Hash Centers
The core of SHC is the generation of hash centers that preserve these semantic relationships while maintaining sufficient separation. We formulate an optimization problem with two primary objectives:
- Semantic Preservation: The Hamming distance between any two hash centers should be inversely proportional to their semantic similarity.
- Minimum Distance Constraint: To prevent "crowding" and ensure distinctiveness, we enforce a minimum distance between centers based on the Gilbert-Varshamov bound.
Solving this NP-hard problem is achieved through the Augmented Lagrangian Method (ALM), which decomposes the global objective into manageable sub-tasks, resulting in a set of optimized binary centers that mirror the semantic geometry of the data.
3. Deep Hashing Network Training
With the semantic centers established, we train a deep hashing network to map input images to these targets. The training utilizes a dual-loss function: a central similarity loss to pull image features toward their respective semantic centers, and a quantization loss to minimize the error when converting continuous features into binary bits.
Experimental Evaluation
We evaluated SHC on five versions of three major datasets: CIFAR-100, Stanford Cars (Versions A & B), and NABirds (Versions A & B). The latter is particularly challenging due to its fine-grained nature, containing 555 classes of birds where semantic relationships are highly complex.
SHC consistently outperformed nine state-of-the-art baselines, including CSQ, MDS, and OrthoHash, across all code lengths (16, 32, and 64 bits). The performance gains were substantial:
- MAP@100: +7.26% average improvement.
- MAP@1000: +7.62% average improvement.
- MAP@ALL: +11.71% average improvement.
The most significant improvements were observed in the NABirds dataset. In fine-grained scenarios where categories are visually similar, the semantic awareness of SHC allows the model to maintain a structured Hamming space that prevents different bird species from collapsing into the same hash region.
Discussion and Insights
A key finding of this research is that the geometry of Hamming space should mirror the geometry of the semantic space. Previous assumptions that maximizing distance between all centers is always optimal are challenged by SHC’s success. By allowing semantically related classes to reside closer together, the model creates a more intuitive and discriminative feature space.
Furthermore, the use of the Gilbert-Varshamov bound proves essential in low-dimensional settings. When the number of classes is high (e.g., 555 in NABirds), the available Hamming space becomes crowded. SHC’s optimization ensures that even while preserving similarity, the centers remain distinct enough to avoid misclassification.
Conclusion
The SHC framework demonstrates that incorporating semantic relationships into the generation of hash centers significantly enhances image retrieval performance. By moving away from data-independent center assignment and toward a data-dependent, optimized approach, SHC provides a robust solution for large-scale, fine-grained retrieval tasks. Future work will explore dynamic center evolution during training and the extension of semantic centers to unsupervised learning environments.
Summary updated