Unsupervised Few-Bits Semantic Hashing with Implicit Topics Modeling

Fanghua Ye, Jarana Manotumruksa, Emine YilmazEMNLP 2020 · OpenReview

Introduction

Semantic hashing is a critical technique for mapping high-dimensional text data into low-dimensional binary spaces. This transformation enables lightning-fast similarity searches, as computers can compare binary strings using Hamming distance significantly faster than they can process raw text or dense embeddings. However, traditional semantic hashing faces a significant "Information Loss" bottleneck. When documents are compressed into extremely short codes (e.g., 4 to 12 bits), standard models struggle to retain sufficient semantic information for accurate retrieval. Most existing research focuses on longer codes (32+ bits), leaving the "few-bits" regime—essential for massive-scale short-text data like tweets or reviews—relatively under-studied.

The WISH Framework

To address this bottleneck, we introduce WISH (feW-bIts Semantic Hashing), a generative framework built on the Variational Autoencoder (VAE) architecture. The core innovation of WISH lies in its treatment of hash bits as "semantic switches" rather than mere data points.

The WISH architecture: Binary hash bits z act as selectors for continuous topic vectors t_i to reconstruct the document.

Implicit Topic Modeling

Instead of attempting to reconstruct a document directly from a sparse binary code $z$, WISH treats each bit in the hash as a selector for a corresponding continuous "topic vector" $t_i$. If the $i$-th bit is 1, the associated topic vector is activated. The generative network then reconstructs the text based on the combination of these activated vectors. This allows a tiny 4-bit code to effectively "point" to complex combinations of four distinct, rich semantic themes, bypassing the traditional information bottleneck.

Training and Constraints

To ensure the model is both effective and interpretable, several key mechanisms are employed:

  • Straight-Through Estimator: This allows the binary bottleneck to be differentiable, enabling the model to learn hash codes and topic vectors simultaneously via backpropagation.
  • Orthogonal Constraint: A mathematical penalty is added to the training objective to force learned topic vectors to be as independent as possible, ensuring each bit represents a unique, non-redundant topic.
  • Stochastic Sampling: The use of stochastic multivariate Bernoulli variables during training encourages the model to explore a wider range of semantic representations.

Experimental Evaluation

WISH was evaluated on four standard text benchmarks: Reuters (news documents), TMC (air traffic reports), 20Newsgroups (forum posts), and Agnews (large-scale news articles). Performance was measured using Precision@100, which calculates the proportion of the top 100 retrieved results that share a label with the query.

Performance comparison (Prec@100) between LDA and WISH across four standard text benchmarks.

Key Results

  • Superiority in Few-Bits: WISH consistently outperformed state-of-the-art baselines across all datasets. For instance, on the 20Newsgroups dataset with a 4-bit code, WISH achieved a precision of 0.3543, nearly 10% higher than the next best baseline.
  • Scalability and Interpretability: While optimized for few-bits, WISH remains competitive at higher bit lengths (up to 32 bits). Furthermore, because each bit corresponds to a specific topic vector, the resulting hash codes are inherently interpretable—one bit might consistently represent "Sports" while another represents "Finance."
  • Clustering Quality: Visualization using UMAP revealed that WISH creates much tighter, more distinct document clusters compared to previous methods, which often produced overlapping or noisy representations.

Conclusion and Future Work

WISH demonstrates that treating hash bits as selectors for implicit topics can significantly mitigate information loss in ultra-compact semantic hashing. Future work may explore more sophisticated integration strategies for topic vectors, such as attention mechanisms, or the incorporation of semi-supervision to further refine topic representations for specific domains. Currently, WISH stands as a robust solution for efficient, large-scale retrieval in resource-constrained or high-velocity text environments.

Summary updated

BibTeX
@inproceedings{ye2020unsupervised,
  author = {Fanghua Ye and Jarana Manotumruksa and Emine Yilmaz},
  title = {{Unsupervised Few-Bits Semantic Hashing with Implicit Topics Modeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=0YEy4-XTtg},
}

Loading related content...

Figure