Abstract
Recently, discrete latent variable models have received a surge of interest in both Natural Language Processing (NLP) and Computer Vision (CV), attributed to their comparable performance to continuous counterparts in representation learning while being more interpretable. In this paper, we develop a topic-informed discrete latent variable model for semantic textual similarity (STS), which learns a shared latent space for sentence-pair representation via vector quantization. Compared with previous models limited to local semantic contexts, our model explores richer semantic information via topic modeling. We further boost performance by injecting the quantized representation into a transformer-based language model with a well-designed semantic-driven attention mechanism. Extensive experiments across various English language datasets demonstrate that our model surpasses several strong neural baselines in STS tasks.
Introduction
Semantic Textual Similarity (STS) is a fundamental task in NLP, requiring models to determine the degree of semantic equivalence between sentence pairs. While state-of-the-art Transformer models like BERT and RoBERTa have set high benchmarks, they primarily rely on continuous latent variables. These mathematical representations, while effective, often function as "black boxes," offering limited interpretability. Furthermore, standard Transformer architectures typically focus on local semantic contexts—the relationships between words within a specific sentence—often overlooking the "global" semantics or broader thematic structures present across a dataset.
To address these challenges, this research introduces a framework that combines the global context of Neural Topic Models (NTM) with the interpretability of discrete latent variables via Vector-Quantized Variational Autoencoders (VQ-VAE). By integrating these "topic-informed" discrete representations into Transformers through a custom attention mechanism, the authors propose two enhanced models: DisBert and DisRoBERTa.
Methodology
The proposed architecture consists of three integrated components designed to bridge the gap between local and global semantic understanding:
1. Neural Topic Component (NTM)
This module learns latent "topics" (distributions of words) from the entire corpus. By analyzing the dataset as a whole, the NTM provides a global semantic map that identifies thematic relationships that might not be apparent within a single sentence pair.
2. Topic-enhanced VQ-VAE
Discrete latent variables are theoretically well-suited to the discrete nature of language, but training them via Vector Quantization (VQ) is notoriously difficult. A common failure mode is "codebook collapse," where the model utilizes only a small subset of available representations. This framework mitigates this by using the topics learned by the NTM to initialize the discrete codebook. This "topic-informed" initialization ensures that discrete variables have clear semantic grounding from the start, stabilizing the training process.
3. Semantics-driven Multi-head Attention
This "plug-and-play" module allows the discrete representations from the VQ-VAE to influence the Transformer’s internal processing. It calculates attention scores by considering global topic relevance, effectively instructing the model to prioritize words that belong to similar global topics when comparing two sentences.
Experimental Setup and Results
The models were evaluated across six diverse English language datasets, including the Microsoft Research Paraphrase Corpus (MRPC), Quora Question Pairs, and the Semantic Textual Similarity Benchmark (STS-B).
Performance Gains
The "Dis-" variants consistently outperformed their vanilla counterparts:
- DisBert showed improvements ranging from 0.1% to 5.5% over standard BERT across all benchmarks.
- DisRoBERTa achieved state-of-the-art results, notably reaching an F1 score of 91.15 on MRPC and 91.81 on Quora.
- On STS-B, the model achieved a Spearman correlation of 89.28, outperforming other topic-aware baselines like tBERT.
Data Efficiency and Training Dynamics
One of the most significant findings was the model's data efficiency. DisBert outperformed standard BERT even when trained on only 20% of the available data. This suggests that global topic information acts as a powerful prior that helps the model generalize more effectively when examples are scarce. Additionally, the researchers found that Joint Training (training the NTM and VQ-VAE simultaneously) yielded superior results compared to multistage training, as the components learned to optimize their representations in tandem.
Discussion and Limitations
While the model provides a significant boost in performance and interpretability, it introduces certain trade-offs:
- Computational Overhead: The integration of three distinct components (NTM, VQ-VAE, and Transformer) increases training time. DisBert takes approximately three times longer to train than a standard BERT model.
- Scalability of Gains: The benefits of global topic information are most pronounced on smaller datasets and shorter sentences. As sentence length increases or datasets become massive, the relative advantage of adding global context diminishes, as the local context within the Transformer becomes increasingly sufficient.
Conclusion
By combining Neural Topic Modeling with discrete latent variables, this framework provides a more nuanced and interpretable approach to Semantic Textual Similarity. The transition from continuous "black box" representations to topic-informed discrete variables allows models like DisBert and DisRoBERTa to capture global thematic nuances that standard Transformers miss, leading to improved performance and superior data efficiency in low-resource scenarios.
Summary updated