Accurate Deep Representation Quantization with Gradient Snapping Layer for Similarity Search

Shicong Liu, Hongtao LuarXiv 2016 · OpenReview

Abstract

Recent advances in large-scale similarity search involve using deeply learned representations to improve search accuracy and vector quantization methods to increase search speed. However, learning deep representations that strongly preserve similarities while remaining accurately quantizable remains challenging. Existing methods often leverage quantization loss and similarity loss, which can result in biased back-propagating gradients that degrade performance. We propose a novel Gradient Snapping Layer (GSL) to directly regularize the back-propagating gradient towards a neighboring codeword. The generated gradients are unbiased for reducing similarity loss and propel the learned representations to be accurately quantized. The proposed framework is compatible with various vector quantization approaches and demonstrates state-of-the-art performance across multiple benchmarks.

Introduction

In the domain of large-scale similarity search—such as finding similar images in databases containing millions of entries—researchers face a fundamental trade-off between accuracy and speed. High accuracy is typically achieved through complex feature vectors (deep representations) learned by Convolutional Neural Networks (CNNs). High speed, conversely, is achieved through quantization, which compresses these high-dimensional vectors into short, discrete codes for rapid comparison.

The primary challenge is that deep learning and quantization are often at odds. Standard approaches attempt to force networks to produce "quantizable" features by adding a quantization loss term to the training objective. This often creates a conflict: the similarity loss tries to cluster semantically related images, while the quantization loss pulls features toward arbitrary codebook centers. This conflict results in sub-optimal features that lose discriminative power.

The Gradient Snapping Layer (GSL)

The core innovation of this work is the Gradient Snapping Layer (GSL), a plug-and-play component that modifies the learning process itself rather than just the loss function.

Overview of the proposed framework featuring the Gradient Snapping Layer (GSL) integrated into a deep neural network.

Snapping vs. Penalizing

Unlike traditional methods that add a penalty term, GSL changes how the network updates during back-propagation:

  • Forward Pass: The GSL acts as an identity layer, passing deep features through unchanged to ensure no information is lost during the initial inference.
  • Backward Pass: When the gradient signal (derived from similarity losses like triplet or contrastive loss) returns, the GSL "snaps" it. It identifies neighboring codewords in the quantization codebook and adjusts the gradient direction. This ensures the network learns to move features toward codewords without distorting the original semantic similarity signal.

Mechanism and Intuition

By directly regularizing the gradient, the network avoids the "biased" updates common in multi-task loss functions. The GSL aligns the learning signal with the underlying quantization structure, allowing for joint optimization where the network learns better features while the quantization codebook is periodically refined (e.g., via k-means).

Visual intuition of the gradient snapping process, showing how gradients are redirected toward codewords to facilitate better quantization.

A key feature of this mechanism is the "rejection" capability. If the similarity gradient and the codeword direction are diametrically opposed, the GSL can choose to reject the snapping. This prevents the quantization process from destroying the semantic meaning of a feature just for the sake of compression.

Experimental Results

The GSL framework was evaluated on four major benchmarks: CIFAR-10, MNIST, NUS-WIDE, and ImageNet (ILSVRC2012). It consistently outperformed existing state-of-the-art deep hashing and quantization methods:

  • CIFAR-10: Using 32-bit codes, GSL achieved a Mean Average Precision (MAP) of 0.747, significantly higher than the Deep Quantization Network (DQN) at 0.564.
  • NUS-WIDE: GSL reached an MAP of 0.839 (32-bit), maintaining a substantial lead over competing methods like DHN (0.748).
  • Scalability: On ImageNet, the method demonstrated robust retrieval performance, proving that local gradient snapping leads to superior global representations in large-scale settings.

Conclusion and Future Work

The Gradient Snapping Layer provides a flexible, modular solution for bridging the gap between deep feature learning and efficient vector quantization. While the current implementation introduces some computational overhead due to neighbor searching in the codebook, its ability to produce unbiased, quantizable gradients represents a significant step forward for similarity search. Future research may explore more sophisticated rejection mechanisms and the application of GSL to modern transformer-based architectures.

Summary updated

Abstract

Recent advance of large scale similarity search involves using deeply learned representations to improve the search accuracy and use vector quantization methods to increase the search speed. However, how to learn deep representations that strongly preserve similarities between data pairs and can be accurately quantized via vector quantization remains a challenging task. Existing methods simply leverage quantization loss and similarity loss, which result in unexpectedly biased back-propagating gradients and affect the search performances. To this end, we propose a novel gradient snapping layer (GSL) to directly regularize the back-propagating gradient towards a neighboring codeword, the generated gradients are un-biased for reducing similarity loss and also propel the learned representations to be accurately quantized. Joint deep representation and vector quantization learning can be easily performed by alternatively optimize the quantization codebook and the deep neural network. The proposed framework is compatible with various existing vector quantization approaches. Experimental results demonstrate that the proposed framework is effective, flexible and outperforms the state-of-the-art large scale similarity search methods.
BibTeX
@misc{liu2016accurate,
  author = {Shicong Liu and Hongtao Lu},
  title = {{Accurate Deep Representation Quantization with Gradient Snapping Layer for Similarity Search}},
  year = {2016},
  eprint = {1610.09645},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1610.09645},
}
Figure