Measuring semantic similarity—determining if two sentences mean the same thing—is a cornerstone of natural language processing (NLP), powering applications from chatbots to search engines. Historically, researchers have relied on two primary approaches, both with notable limitations:
- Word Mover’s Distance (WMD): This method treats sentences as "bags of words" and calculates the minimum cost to transform one bag into another. While accurate, it is computationally expensive and ignores word order.
- Sentence Embeddings: These map sentences to single, static vectors. While fast, they often struggle to capture the inherent ambiguity or "uncertainty" of human language.
To bridge this gap, this paper introduces a Variational Siamese framework. Instead of static vectors, this model represents sentences as Gaussian distributions (intents). By replacing the computationally expensive discrete Word Mover’s Distance with a continuous Wasserstein-2 metric between these distributions, the model achieves state-of-the-art performance in detecting duplicate questions while remaining highly efficient for real-time retrieval.
Methods: The Variational Siamese Network
The core innovation of this approach unfolds in two main stages: generative pretraining and discriminative learning.
1. Generative Pretraining (Learning "Intents")
The model utilizes a Variational Autoencoder (VAE) architecture to learn meaningful sentence representations. A bidirectional LSTM (Bi-LSTM) encoder takes an input sentence and maps it to a Gaussian distribution, defined by a mean ($\mu$) and a diagonal covariance matrix ($\sigma^2$).
To ensure these distributions capture true semantic meaning, the model is trained on a "Repeat/Reformulate" task. It must reconstruct either the original sentence (repeat) or a known paraphrase (reformulate) from a point sampled within that distribution using an LSTM decoder. Intuitively, the mean represents the core "intent" of the question, while the variance captures its linguistic flexibility or ambiguity.
2. Discriminative Learning (The Siamese Framework)
Once the model can reliably represent sentences as distributions, it employs a Siamese network to compare them. Given two questions, the model calculates the Wasserstein-2 distance between their respective Gaussian distributions. Because the covariance matrices are diagonal, this distance has a simple, closed-form solution that is exceptionally fast to compute.
The model then feeds this continuous distance metric—along with a Hadamard product of the means—into a Multi-Layer Perceptron (MLP) to predict whether the sentence pair constitutes a duplicate.
Key Insights and Results
The model was evaluated on the Quora Question Pairs dataset, a standard benchmark containing over 404,000 sentence pairs annotated as either "duplicate" or "not duplicate."
- Uncertainty as a Feature: By utilizing distributions instead of single points, the model explicitly accounts for the multiple interpretations a single sentence can have. The variance term ($\sigma$) acts as a buffer for linguistic noise.
- The Power of Reformulation: Training the model to generate a paraphrase (reformulate) provides a significantly better initialization for similarity tasks than simply training it to reconstruct the input (repeat). It forces the latent space to prioritize underlying semantic meaning over exact word choice.
- High Accuracy and Efficiency: The "VAR-Siamese" model achieved a test accuracy of 88.86% on the Quora dataset. It outperformed many complex models and remained competitive with "joint-encoding" models that read both sentences together. Crucially, because it is a "read-separately" model, sentence representations can be pre-computed, allowing it to perform over 3,500 comparisons per second on a pair of Tesla K80 GPUs.
- Ablation: Without the generative pretraining step, the model's accuracy plummeted to approximately 62%, proving that the "repeat/reformulate" task is essential for properly organizing the latent semantic space.
Limitations and Future Work
To maintain computational efficiency, the author utilized diagonal covariance matrices. However, this assumes that different dimensions of the "intent" are independent, which may not fully reflect the complexities of human language. Future work could explore non-diagonal matrices to capture more intricate linguistic patterns.
Additionally, while the current work focuses on binary similarity (duplicate vs. not duplicate), the framework could be extended to Natural Language Inference (NLI) tasks, where the model must determine entailment, contradiction, or neutrality. Scaling the pretraining to even larger datasets, such as PARANMT-50M, could also further refine the continuous latent space.
Summary updated