MLS3RDUH: Deep Unsupervised Hashing via Manifold based Local Semantic Similarity Structure Reconstructing

Rong-Cheng Tu, Xianling Mao, Wei Wei

Abstract

Efficient image retrieval in large-scale datasets relies on hashing—mapping high-dimensional data to compact binary codes. In unsupervised settings, defining similarity is challenging because visual proximity in feature space does not always imply semantic similarity. We present MLS3RDUH, a deep unsupervised hashing framework that leverages manifold learning to filter out "noisy" neighbors. By reconstructing a local semantic structure and employing a novel log-cosh loss function, MLS3RDUH achieves state-of-the-art performance across multiple benchmarks, significantly improving retrieval accuracy without manual labels.

Introduction: The "Noisy Neighbor" Problem

In unsupervised hashing, the standard approach is to assume that an image's $k$-nearest neighbors in a feature space (e.g., using cosine similarity) are semantically related. However, this assumption often fails because visual similarity is not equivalent to semantic similarity.

Visual similarity vs. semantic similarity: A bird (a) and an airplane (b) can appear visually similar due to background and lighting, despite belonging to different categories.

As shown above, a bird and an airplane might share similar pixel distributions when silhouetted against a sunset. A naive algorithm would treat them as similar, introducing "noise" that degrades the quality of the learned hash codes. MLS3RDUH addresses this by identifying and filtering these imposters using the intrinsic manifold structure of the data.

Proposed Method: MLS3RDUH

The MLS3RDUH framework consists of two primary stages: Similarity Generating and the Hashing Network.

The MLS3RDUH framework: (Top) Generating a refined similarity matrix using manifold filtering. (Bottom) Training a hashing network with the novel Log-Cosh loss.

1. Similarity Matrix Generating

To move beyond simple distance metrics, the model employs Manifold Filtering. By performing a random walk on a neighbor graph, the system identifies the underlying manifold structure. A pair of images is only considered "truly similar" if they are both close in feature space and reside on the same manifold. This process effectively separates semantic neighbors from visually similar but semantically distinct "noise."

Furthermore, the method adopts a tri-modal similarity approach. Instead of binary 0/1 labels, it uses:

  • 1 (Similar): High-ranked neighbors on the same manifold.
  • -1 (Dissimilar): Neighbors flagged as "noisy" or distant points.
  • Continuous range [-1, 1] (Fuzzy): For pairs with ambiguous relationships, providing a more nuanced learning signal.

2. Hashing Network and Log-Cosh Loss

The hashing network uses a deep CNN (based on AlexNet) to map images to binary codes. To optimize this, the authors introduce a log-cosh hashing loss. Since the discrete "sign" function used for binarization is non-differentiable, the log-cosh function provides a smooth, stable approximation. This allows the network to effectively minimize the discrepancy between the learned codes and the refined similarity matrix.

Experimental Results

The researchers evaluated MLS3RDUH on three standard benchmarks: NUS-WIDE, MS COCO, and CIFAR10. In all tests, the labels were reserved strictly for evaluation.

Key findings include:

  • State-of-the-Art Performance: On the NUS-WIDE dataset (64-bit), MLS3RDUH achieved a Mean Average Precision (MAP) of 0.750, outperforming the previous best method, DistillHash, by 7.9%.
  • Ablation Evidence: Removing the manifold reconstruction component caused the MAP on NUS-WIDE to drop from 0.750 to 0.687, confirming that manifold filtering is critical for handling noisy data.
  • Efficiency: The resulting binary codes are more compact and semantically meaningful, leading to faster and more reliable image searches in real-world applications.

Conclusion

MLS3RDUH demonstrates that manifold learning is a powerful tool for cleaning the noise inherent in unsupervised feature spaces. By distinguishing between visual and semantic similarity, the model reconstructs a more accurate local structure for hashing. Future work may focus on reducing the computational overhead of manifold construction to scale the method to billions of images.

Summary updated

Abstract

Most of the unsupervised hashing methods usually map images into semantic similarity-preserving hash codes by constructing local semantic similarity structure as guiding information, i.e., treating each point similar to its k nearest neighbours. However, for an image, some of its k nearest neighbours may be dissimilar to it, i.e., they are noisy datapoints which will damage the retrieval performance. Thus, to tackle this problem, in this paper, we propose a novel deep unsupervised hashing method, called MLS3RDUH, which can reduce the noisy datapoints to further enhance retrieval performance. Specifically, the proposed method first defines a novel similarity matrix by utilising the intrinsic manifold structure in feature space and the cosine similarity of datapoints to reconstruct the local semantic similarity structure. Then a novel log-cosh hashing loss function is used to optimize the hashing network to generate compact hash codes by incorporating the defined similarity as guiding information. Extensive experiments on three public datasets show that the proposed method outperforms the state-of-the-art baselines.
BibTeX
@inproceedings{tu2020mls,
  author = {Rong-Cheng Tu and Xianling Mao and Wei Wei},
  title = {{MLS3RDUH: Deep Unsupervised Hashing via Manifold based Local Semantic Similarity Structure Reconstructing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/479},
}
Figure