Doc2hash: Learning Discrete Latent variables for Documents Retrieval

Yifei Zhang, Hao ZhuNAACL 2019 · OpenReview

Abstract

Doc2hash: Learning Discrete Latent variables for Documents Retrieval. Yifei Zhang, Hao Zhu. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

2019.

Introduction

Efficient document retrieval is a critical component of modern information systems. Semantic hashing aims to map high-dimensional text documents into compact binary "hash codes" where similar documents are assigned codes with small Hamming distances. This allows for rapid retrieval across massive datasets by calculating Hamming distances in milliseconds. However, training deep learning models to produce these discrete codes is challenging because discrete operations (e.g., thresholding) are non-differentiable, preventing standard backpropagation.

Previous approaches often relied on Variational Autoencoders (VAEs) with "Straight-Through" (ST) estimators. While functional, ST estimators introduce significant bias and variance, often resulting in sub-optimal, imbalanced hash codes. This paper introduces Doc2hash, a generative framework that utilizes a continuous Gumbel-Softmax relaxation to enable true end-to-end training of discrete latent variables for document retrieval.

Methodology

Doc2hash treats the hashing process as learning discrete latent variables within a Neural Variational Inference (NVI) framework. The core innovation is the replacement of hard discrete layers with a differentiable approximation.

Gumbel-Softmax Relaxation

To overcome the non-differentiability of discrete nodes, the authors employ the Gumbel-Softmax trick. Instead of a hard argmax function, they use a "tempting softmax" relaxation. This creates a differentiable approximation of the discrete layer, allowing gradients to flow smoothly during training.

Figure 1: The Doc2hash architecture utilizing Gumbel-Softmax relaxation for learning discrete latent variables.

Training and Gradient Flow

By using this relaxation, the model maintains a differentiable path for gradients during the backward pass. This contrasts with traditional discrete latent variable models where gradients cannot propagate through stochastic nodes, necessitating biased estimators like ST.

Figure 2: Comparison of gradient flow in (1) relaxed differentiable nodes vs (2) traditional stochastic discrete nodes.

The framework also incorporates categorical priors to better align with the discrete nature of the task and can be extended to a supervised setting by adding a discriminative loss that ensures hash codes are predictive of document labels.

Experiments and Results

The model was evaluated on three standard benchmarks: TMC (air traffic reports), Reuters (news articles), and RCV1 (large-scale newswire stories). Performance was measured using the precision of the top 100 retrieved documents across various bit lengths (8 to 128 bits).

Performance Metrics

Doc2hash consistently outperformed state-of-the-art baselines. In unsupervised settings on the TMC dataset, Doc2hash achieved a precision of 0.7595 at 128 bits, a significant improvement over the previous state-of-the-art, NASH-DN (0.6325). In supervised settings on RCV1, the model reached near-perfect precision (0.9810 at 32 bits), outperforming supervised versions of VDSH and KSH.

Coding Efficiency and the Entropy Advantage

A key finding was the "Entropy Advantage." Efficient hashing requires a balanced distribution of bits (near 50% ones and zeros). The authors found that the Gumbel-Softmax approach naturally produced a much more balanced distribution (approx. 49% ones) compared to the Straight-Through estimator (approx. 40% ones), leading to higher coding efficiency and information density.

Conclusion

Doc2hash demonstrates that learning binarization during the training process via continuous relaxation leads to superior document representations. The framework is robust to varying bit lengths and produces semantically rich, well-balanced hash codes. While finding the optimal annealing schedule for the temperature parameter gamma remains a practical challenge, the empirical results suggest that Doc2hash sets a new standard for discrete latent variable models in text retrieval.

Summary updated

BibTeX
@inproceedings{zhang2019doc,
  author = {Yifei Zhang and Hao Zhu},
  title = {{Doc2hash: Learning Discrete Latent variables for Documents Retrieval}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=HyNmh7ZuWH},
}
Figure