Neighborhood Preserving Hashing for Scalable Video Retrieval

Shuyan Li, Zhixiang Chen, Jiwen Lu, Xiu Li, Jie Zhou

Abstract

Neighborhood Preserving Hashing (NPH) is an unsupervised framework designed for efficient video retrieval. By leveraging a neighborhood attention mechanism guided by spatial-temporal anchors, NPH selectively encodes relevant video content into compact binary codes. Experimental results on FCVID, YFCC, and ActivityNet demonstrate that NPH significantly outperforms state-of-the-art unsupervised hashing methods, achieving up to a 70% improvement in retrieval accuracy.

1. Introduction

Scalable video retrieval—the task of finding similar videos in massive databases—increasingly relies on hashing techniques to convert high-dimensional video data into short, searchable binary codes. The primary objective is to maintain a low Hamming distance between codes of semantically similar videos. However, existing deep hashing methods often struggle with indiscriminate compression, treating redundant or noisy frames with the same weight as critical content. Furthermore, the heavy reliance on manual labels in supervised methods limits their applicability to web-scale datasets.

2. Methodology

The core of the NPH framework is the Neighborhood Attention Mechanism, which guides the encoding process using the video's broader context.

2.1 Neighborhood Representation and Selective Encoding

To avoid the computational cost of full K-nearest neighbor graphs, NPH utilizes K-means clustering to establish a set of spatial-temporal "anchors." A video's neighborhood is represented by its nearest anchors. This information is injected into the initial step of an RNN-based encoder, acting as a filter that prioritizes visual features aligned with the neighborhood context while suppressing irrelevant background noise.

The NPH architecture featuring the neighborhood attention mechanism and multi-task loss functions.

2.2 Multi-Task Learning Objectives

The model is optimized using three simultaneous loss functions:

  1. Neighborhood Similarity Loss ($L_s$): Preserves the similarity structure of the original feature space within the binary Hamming space.
  2. Visual Content Reconstruction Loss ($L_{vr}$): Ensures the binary code retains sufficient information to reconstruct the original video frames.
  3. Neighborhood Info Reconstruction Loss ($L_{nr}$): Verifies that the encoder's final hidden state effectively captured the initial neighborhood guidance.

3. Experimental Evaluation

The authors evaluated NPH on three large-scale benchmarks:

  • FCVID (Fudan-Columbia Video Dataset): Over 91,000 web videos across 239 categories.
  • YFCC (Yahoo Flickr Creative Commons): A massive collection where the authors used ~400,000 unlabeled videos for training.
  • ActivityNet: Approximately 20,000 videos focused on human activities.

4. Results and Discussion

NPH consistently outperformed state-of-the-art unsupervised methods, including SSTH, JTAE, and SSVH, across all tested bit lengths (8, 16, 32, and 64 bits).

Comprehensive retrieval performance (mAP) comparison across three datasets and four bit lengths.

4.1 Performance Gains

On the FCVID dataset with 16-bit codes, NPH achieved a 70% improvement in mean Average Precision (mAP) over the previous best method, SSVH. Precision-Recall curves further indicate that NPH maintains superior precision across various recall levels. Qualitative analysis shows NPH's ability to distinguish between visually similar but semantically distinct actions, such as "Yoga" versus "Making Shorts," by ignoring distracting background similarities.

4.2 Key Insights

The success of NPH highlights the value of context-aware hashing. By providing the model with a "hint" of its neighbors via fixed anchors, the unsupervised learner can effectively prioritize salient features. This neighborhood attention mechanism is architecture-agnostic and could potentially be integrated into Transformers or other advanced neural structures.

5. Conclusion

NPH presents a robust solution for unsupervised video hashing. While performance on ActivityNet suggests challenges in uncontrolled environments with smaller retrieval sets, the framework's efficiency and accuracy make it a promising tool for large-scale applications. Future work may explore dynamic neighborhood optimization and the integration of NPH with hierarchical deep learning architectures.

Summary updated

Abstract

In this paper, we propose a Neighborhood Preserving Hashing (NPH) method for scalable video retrieval in an unsupervised manner. Unlike most existing deep video hashing methods which indiscriminately compress an entire video into a binary code, we embed the spatial-temporal neighborhood information into the encoding network such that the neighborhood-relevant visual content of a video can be preferentially encoded into a binary code under the guidance of the neighborhood information. Specifically, we propose a neighborhood attention mechanism which focuses on partial useful content of each input frame conditioned on the neighborhood information. We then integrate the neighborhood attention mechanism into an RNN-based reconstruction scheme to encourage the binary codes to capture the spatial-temporal structure in a video which is consistent with that in the neighborhood. As a consequence, the learned hashing functions can map similar videos to similar binary codes. Extensive experiments on three widely-used benchmark datasets validate the effectiveness of our proposed approach.
BibTeX
@inproceedings{li2019neighborhood,
  author = {Shuyan Li and Zhixiang Chen and Jiwen Lu and Xiu Li and Jie Zhou},
  title = {{Neighborhood Preserving Hashing for Scalable Video Retrieval}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00830},
}
Figure