Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables

Erxin Yu, Lan Du, Yuan Jin, Zhepei Wei, Yi Chang

Abstract

This paper introduces a novel framework for Semantic Textual Similarity (STS) that integrates global topic modeling with discrete latent variables. By guiding a Vector-Quantized Variational Autoencoder (VQ-VAE) with a Neural Topic Model (NTM) and incorporating the resulting discrete representations into Transformer architectures via a specialized attention mechanism, the proposed DisBert and DisRoBERTa models achieve state-of-the-art performance. The approach addresses the limitations of continuous representations and local context focus in standard Transformers, while mitigating "codebook collapse" in VQ-VAEs through topic-informed initialization.

Introduction

Semantic Textual Similarity (STS) is the task of determining the degree of semantic proximity between two pieces of text. While modern deep learning models like BERT and RoBERTa have established high benchmarks, they face two primary challenges. First, most Transformers focus on "local" context within a sentence, often missing "global" semantics that relate words across an entire corpus. Second, natural language is inherently discrete, yet most models rely on continuous representations. Discrete models like VQ-VAEs exist but are prone to "codebook collapse," where only a small fraction of the representational capacity is utilized. This work proposes a hybrid architecture that leverages discrete topic-informed latent variables to enhance Transformer-based STS models.

Methodology

The authors propose a three-part architecture to bridge the gap between local and global context:

  1. Neural Topic Model (NTM): This component learns global semantics by treating documents as distributions over topics, providing a macro-level view of the vocabulary.
  2. Topic-enhanced VQ-VAE: Instead of learning arbitrary clusters, this model uses the NTM’s topic distributions to inform its encoder. Crucially, it initializes its "codebook" (discrete latent vectors) using topic embeddings from the NTM, ensuring the discrete variables have clear semantic meaning and preventing codebook collapse.
  3. Semantics-driven Multi-head Attention: This "plug-and-play" module injects the quantized, topic-informed representations into the Transformer’s self-attention layers, allowing the model to attend to global topic similarities when calculating sentence relevance.
Figure 1: Overview of the proposed framework, including the Neural Topic Model, Topic-enhanced VQ-VAE, and Semantics-attended Transformer.

Experiments and Datasets

The model was evaluated on six prominent English language datasets for semantic matching and similarity:

  • MRPC: Microsoft Research Paraphrase Corpus.
  • Quora: The Quora Question Pairs dataset.
  • STS-B: The Semantic Textual Similarity Benchmark.
  • SemEval CQA: Three subtasks (A, B, and C) from the Community Question Answering track.

Results and Discussion

DisBert and DisRoBERTa consistently outperformed strong baselines, including vanilla BERT/RoBERTa, ERNIE, and tBERT. On the MRPC dataset, DisBert improved F1 scores by approximately 0.8% over standard BERT. On the challenging SemEval-C subtask, DisRoBERTa achieved a significant lead over the standard RoBERTa baseline.

Figure 3: Ablation study results on MRPC and Quora datasets, showing the impact of removing various components of the DisBert/DisRoBERTa framework.

Ablation studies (Figure 3) demonstrate that removing the topic-informed initialization or the custom attention mechanism leads to measurable performance drops, confirming that the integration of global topics is the primary driver of success. Furthermore, the model showed significant robustness to data scarcity, with DisBert outperforming standard BERT even when trained on only 20% of available data.

Conclusion and Future Work

The integration of global topic information via discrete latent variables significantly improves STS performance and model interpretability. By tying latent variables to topics, the model can provide explanations for similarity based on shared thematic clusters. Future work will focus on reducing the computational overhead—currently roughly 3x that of a standard BERT—and exploring methods to dynamically determine the optimal number of topics for a given corpus.

Summary updated

Abstract

Recently, discrete latent variable models have received a surge of interest in both Natural Language Processing (NLP) and Computer Vision (CV), attributed to their comparable performance to the continuous counterparts in representation learning, while being more interpretable in their predictions. In this paper, we develop a topic-informed discrete latent variable model for semantic textual similarity, which learns a shared latent space for sentence-pair representation via vector quantization. Compared with previous models limited to local semantic contexts, our model can explore richer semantic information via topic modeling. We further boost the performance of semantic similarity by injecting the quantized representation into a transformer-based language model with a well-designed semantic-driven attention mechanism. We demonstrate, through extensive experiments across various English language datasets, that our model is able to surpass several strong neural baselines in semantic textual similarity tasks.
BibTeX
@inproceedings{yu2022learning,
  author = {Erxin Yu and Lan Du and Yuan Jin and Zhepei Wei and Yi Chang},
  title = {{Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=ydsXUfPAZR},
}
Figure