Deep Hashing with Semantic Hash Centers for Image Retrieval

Li Chen, Rui Liu, Yuxiang Zhou, Xudong Ma, Yong Chen, Dell Zhang

TL;DR

The SHC (Semantic Hash Centers) framework improves large-scale image retrieval by ensuring that semantically related classes (like "cat" and "dog") have hash centers closer together in Hamming space than unrelated ones. By combining data-dependent similarity modeling with rigorous distance constraints, SHC achieves significant performance gains—up to 11.7% in Mean Average Precision—over existing state-of-the-art methods.

What Problem Does This Paper Address?

In the era of big data, searching through millions of images requires converting them into compact binary "hash codes." This allows for lightning-fast retrieval using simple hardware-level XOR operations.

Recent "point-wise" hashing methods (like CSQ and MDS) have improved this process by pre-assigning a "hash center" (a target binary code) to each category. However, these methods generate centers using purely mathematical algorithms (like Hadamard matrices or Bernoulli distributions) that are data-independent. They treat all classes as equally different. In reality, a "Golden Retriever" is semantically closer to a "Labrador" than it is to a "Toaster." Ignoring these semantic relationships leads to hash codes that lack discriminative power, ultimately hurting retrieval accuracy.

Methods

The authors propose a three-stage framework called SHC:

  1. Constructing a Data-Dependent Similarity Matrix: Instead of relying on simple binary labels (same/different), the authors use a pre-trained ResNet34 classification network. By analyzing the "confusion" or probability distributions across classes, they capture rich semantic relationships. If the network often sees similarities between two classes, those classes are assigned a higher similarity score.
  2. Generating Semantic Hash Centers: This is the core innovation. The authors developed an optimization algorithm that balances two competing goals:
  3. Semantic Preservation: Keeping the Hamming distance between hash centers proportional to their semantic similarity.
  4. Minimum Distance Constraint: Using the Gilbert-Varshamov bound to ensure that centers don't get too close or overlap, which would cause misclassification.
  5. They solve this NP-hard problem using the Augmented Lagrangian Method (ALM), breaking it down into manageable sub-optimization tasks.
  6. Training the Deep Hashing Network: Finally, a standard deep network is trained to map input images to these newly generated semantic hash centers using a central similarity loss and a quantization loss.

Datasets

The model was evaluated on five curated versions of three widely-used benchmarks:

  • CIFAR-100: 100 classes of general objects.
  • Stanford Cars (Versions A & B): 196 classes of cars. Version B is a more challenging, "real-world" split where training and test sets share no images.
  • NABirds (Versions A & B): 555 classes of birds. This is a fine-grained dataset where semantic relationships are particularly complex.

Key Insights and Ideas

  • Hamming Space is Not Just About Distance: Previous work assumed that maximizing the distance between all centers was always better. SHC proves that relative distance matters more; the "geometry" of the hash centers should mirror the "geometry" of the semantic concepts.
  • Softmax as a Semantic Proxy: A key "aha moment" is realizing that the output of a classifier contains more information than just the final label. The secondary probabilities (the "almost" guesses) provide a footprint of inter-class similarity that is far more adaptive than static text-based embeddings like BERT.
  • The "Crowding" Problem: The paper highlights a trade-off in low-dimensional spaces (e.g., 16-bit codes). When you have many classes (like the 555 in NABirds), the Hamming space becomes "crowded." SHC manages this by strictly enforcing the Gilbert-Varshamov bound, preventing semantic clusters from collapsing into each other.

Key Results

SHC consistently outperformed nine baseline methods (including CSQ, MDS, and OrthoHash) across all datasets and code lengths (16, 32, and 64 bits):

  • Significant MAP Gains: On average, SHC improved Mean Average Precision (MAP) by +7.26% (MAP@100), +7.62% (MAP@1000), and +11.71% (MAP@ALL) over the next best competitors.
  • Fine-Grained Superiority: The performance jump was most notable on the NABirds dataset, proving that semantic centers are vital when dealing with many highly similar categories.
  • Efficiency: Because it uses point-wise supervision (comparing images to centers rather than comparing every image to every other image), it remains computationally efficient for large-scale training.

Limitations and Future Work

  • Dependency on Pre-training: The quality of the semantic centers depends on the initial classification network. If the classifier fails to capture meaningful relationships for a specific niche dataset, the resulting hash centers may be sub-optimal.
  • Static Centers: Currently, the hash centers are generated once before the final training stage. A potential future direction could involve "evolving" the hash centers dynamically during training to better adapt to the backbone's feature learning.
  • Unsupervised Extension: The current work is supervised (requires labels). Extending this "semantic center" logic to unsupervised or self-supervised settings remains an open challenge.

Summary updated

Abstract

Deep hashing is an effective approach for large-scale image retrieval. Current methods are typically classified by their supervision types: point-wise, pair-wise, and list-wise. Recent point-wise techniques (e.g., CSQ, MDS) have improved retrieval performance by pre-assigning a hash center to each class, enhancing the discriminability of hash codes across various datasets. However, these methods rely on data-independent algorithms to generate hash centers, which neglect the semantic relationships between classes and may degrade retrieval performance. This paper introduces the concept of semantic hash centers, building on the idea of traditional hash centers. We hypothesize that hash centers of semantically related classes should have closer Hamming distances, while those of unrelated classes should be more distant. To this end, we propose a three-stage framework, SHC, to generate hash codes that preserve semantic structure. First, we develop a classification network to identify semantic similarities between classes using a data-dependent similarity calculation that adapts to varying data distributions. Second, we introduce an optimization algorithm to generate semantic hash centers, preserving semantic relatedness while enforcing a minimum distance between centers to avoid excessively similar hash codes. Finally, a deep hashing network is trained using these semantic centers to convert images into binary hash codes. Experimental results on large-scale retrieval tasks across several public datasets show that SHC significantly improves retrieval performance. Specifically, SHC achieves average improvements of +7.26%, +7.62%, and +11.71% in MAP@100, MAP@1000, and MAP@ALL metrics, respectively, over state-of-the-art methods.
BibTeX
@article{chen2025deep,
  author = {Li Chen and Rui Liu and Yuxiang Zhou and Xudong Ma and Yong Chen and Dell Zhang},
  title = {{Deep Hashing with Semantic Hash Centers for Image Retrieval}},
  journal = {ACM Trans. Information Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=r7dPCOi2Xn},
}

Loading related content...

Figure