Abstract
This paper introduces a novel framework for Semantic Textual Similarity (STS) that integrates global topic modeling with discrete latent variables. By guiding a Vector-Quantized Variational Autoencoder (VQ-VAE) with a Neural Topic Model (NTM) and incorporating the resulting discrete representations into Transformer architectures via a specialized attention mechanism, the proposed DisBert and DisRoBERTa models achieve state-of-the-art performance. The approach addresses the limitations of continuous representations and local context focus in standard Transformers, while mitigating "codebook collapse" in VQ-VAEs through topic-informed initialization.
Introduction
Semantic Textual Similarity (STS) is the task of determining the degree of semantic proximity between two pieces of text. While modern deep learning models like BERT and RoBERTa have established high benchmarks, they face two primary challenges. First, most Transformers focus on "local" context within a sentence, often missing "global" semantics that relate words across an entire corpus. Second, natural language is inherently discrete, yet most models rely on continuous representations. Discrete models like VQ-VAEs exist but are prone to "codebook collapse," where only a small fraction of the representational capacity is utilized. This work proposes a hybrid architecture that leverages discrete topic-informed latent variables to enhance Transformer-based STS models.
Methodology
The authors propose a three-part architecture to bridge the gap between local and global context:
- Neural Topic Model (NTM): This component learns global semantics by treating documents as distributions over topics, providing a macro-level view of the vocabulary.
- Topic-enhanced VQ-VAE: Instead of learning arbitrary clusters, this model uses the NTM’s topic distributions to inform its encoder. Crucially, it initializes its "codebook" (discrete latent vectors) using topic embeddings from the NTM, ensuring the discrete variables have clear semantic meaning and preventing codebook collapse.
- Semantics-driven Multi-head Attention: This "plug-and-play" module injects the quantized, topic-informed representations into the Transformer’s self-attention layers, allowing the model to attend to global topic similarities when calculating sentence relevance.
Experiments and Datasets
The model was evaluated on six prominent English language datasets for semantic matching and similarity:
- MRPC: Microsoft Research Paraphrase Corpus.
- Quora: The Quora Question Pairs dataset.
- STS-B: The Semantic Textual Similarity Benchmark.
- SemEval CQA: Three subtasks (A, B, and C) from the Community Question Answering track.
Results and Discussion
DisBert and DisRoBERTa consistently outperformed strong baselines, including vanilla BERT/RoBERTa, ERNIE, and tBERT. On the MRPC dataset, DisBert improved F1 scores by approximately 0.8% over standard BERT. On the challenging SemEval-C subtask, DisRoBERTa achieved a significant lead over the standard RoBERTa baseline.
Ablation studies (Figure 3) demonstrate that removing the topic-informed initialization or the custom attention mechanism leads to measurable performance drops, confirming that the integration of global topics is the primary driver of success. Furthermore, the model showed significant robustness to data scarcity, with DisBert outperforming standard BERT even when trained on only 20% of available data.
Conclusion and Future Work
The integration of global topic information via discrete latent variables significantly improves STS performance and model interpretability. By tying latent variables to topics, the model can provide explanations for similarity based on shared thematic clusters. Future work will focus on reducing the computational overhead—currently roughly 3x that of a standard BERT—and exploring methods to dynamically determine the optimal number of topics for a given corpus.
Summary updated