Abstract
Doc2hash: Learning Discrete Latent variables for Documents Retrieval. Yifei Zhang, Hao Zhu. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
2019.
Introduction
Efficient document retrieval is a critical component of modern information systems. Semantic hashing aims to map high-dimensional text documents into compact binary "hash codes" where similar documents are assigned codes with small Hamming distances. This allows for rapid retrieval across massive datasets by calculating Hamming distances in milliseconds. However, training deep learning models to produce these discrete codes is challenging because discrete operations (e.g., thresholding) are non-differentiable, preventing standard backpropagation.
Previous approaches often relied on Variational Autoencoders (VAEs) with "Straight-Through" (ST) estimators. While functional, ST estimators introduce significant bias and variance, often resulting in sub-optimal, imbalanced hash codes. This paper introduces Doc2hash, a generative framework that utilizes a continuous Gumbel-Softmax relaxation to enable true end-to-end training of discrete latent variables for document retrieval.
Methodology
Doc2hash treats the hashing process as learning discrete latent variables within a Neural Variational Inference (NVI) framework. The core innovation is the replacement of hard discrete layers with a differentiable approximation.
Gumbel-Softmax Relaxation
To overcome the non-differentiability of discrete nodes, the authors employ the Gumbel-Softmax trick. Instead of a hard argmax function, they use a "tempting softmax" relaxation. This creates a differentiable approximation of the discrete layer, allowing gradients to flow smoothly during training.
Training and Gradient Flow
By using this relaxation, the model maintains a differentiable path for gradients during the backward pass. This contrasts with traditional discrete latent variable models where gradients cannot propagate through stochastic nodes, necessitating biased estimators like ST.
The framework also incorporates categorical priors to better align with the discrete nature of the task and can be extended to a supervised setting by adding a discriminative loss that ensures hash codes are predictive of document labels.
Experiments and Results
The model was evaluated on three standard benchmarks: TMC (air traffic reports), Reuters (news articles), and RCV1 (large-scale newswire stories). Performance was measured using the precision of the top 100 retrieved documents across various bit lengths (8 to 128 bits).
Performance Metrics
Doc2hash consistently outperformed state-of-the-art baselines. In unsupervised settings on the TMC dataset, Doc2hash achieved a precision of 0.7595 at 128 bits, a significant improvement over the previous state-of-the-art, NASH-DN (0.6325). In supervised settings on RCV1, the model reached near-perfect precision (0.9810 at 32 bits), outperforming supervised versions of VDSH and KSH.
Coding Efficiency and the Entropy Advantage
A key finding was the "Entropy Advantage." Efficient hashing requires a balanced distribution of bits (near 50% ones and zeros). The authors found that the Gumbel-Softmax approach naturally produced a much more balanced distribution (approx. 49% ones) compared to the Straight-Through estimator (approx. 40% ones), leading to higher coding efficiency and information density.
Conclusion
Doc2hash demonstrates that learning binarization during the training process via continuous relaxation leads to superior document representations. The framework is robust to varying bit lengths and produces semantically rich, well-balanced hash codes. While finding the optimal annealing schedule for the temperature parameter gamma remains a practical challenge, the empirical results suggest that Doc2hash sets a new standard for discrete latent variable models in text retrieval.
Summary updated