Abstract
Paraphrasing requires balancing semantic consistency with syntactic diversity. We present SEPARATOR, a model that generates diverse paraphrases by explicitly disentangling a sentence's meaning from its syntactic form. By using a dual-bottleneck architecture—continuous for meaning and discrete for syntax—the model can "re-skin" the intent of a question into entirely new structures without needing external reference templates at test time.
Introduction
Effective paraphrasing involves transforming the surface form of a sentence while preserving its underlying semantic intent. Traditional models often struggle with this balance, either producing trivial changes (parroting) or requiring human-provided exemplars to guide syntactic variation. We address this by introducing SEPARATOR, a Transformer-based encoder-decoder model designed to factorise latent representations into distinct semantic and syntactic components.
Methodology: The SEPARATOR Architecture
The core innovation of SEPARATOR is its dual-bottleneck latent space. Unlike standard Variational Autoencoders (VAEs) that use a single latent vector, our model splits the representation into two distinct paths:
- Semantic Encoding ($z_{sem}$): A continuous space modeled via a VAE. This captures the fluid, high-dimensional nature of meaning.
- Syntactic Encoding ($z_{syn}$): A discrete space modeled via a Vector-Quantized Variational Autoencoder (VQ-VAE). This captures the categorical nature of sentence structures (e.g., "How much does X weigh?" vs. "What is the weight of X?").
To force the model to separate these features, it is trained on triples $(X_{sem}, X_{syn}, Y)$, learning to reconstruct a target $Y$ using the semantic content of $X_{sem}$ and the syntactic structure of $X_{syn}$. At test time, a small classifier predicts valid discrete syntactic codes for a given intent, allowing the model to sample different "syntactic skins" for the same meaning autonomously.
Experiments and Datasets
We evaluated SEPARATOR on two major English question datasets:
- Paralex: A large, noisy dataset scraped from WikiAnswers containing clusters of questions sharing the same answer.
- Quora Question Pairs (QQP): A cleaner dataset containing subjective and conversational questions.
To ensure data quality, we applied language detection and spell-checking to the noisy Paralex corpus before training.
Key Results
SEPARATOR demonstrates a superior tradeoff between semantic preservation and syntactic novelty. It achieved the highest iBLEU scores—a metric that penalizes similarity to the input—on both Paralex (14.84) and QQP (5.84), significantly outperforming baselines like DiPS and SOW/REAP.
Human Evaluation
In side-by-side comparisons, human judges assessed the models based on meaning preservation, dissimilarity to the input, and fluency. SEPARATOR was found to be significantly more diverse than standard VAEs while remaining more fluent and meaning-preserving than other diversity-focused models.
Discussion and Insights
Our analysis reveals that discrete syntax is a natural fit for language structure. While meaning is infinite and nuanced, the number of common ways to structure a question is relatively finite. By analyzing the entropy of the quantizer heads, we found that different heads "learned" specific syntactic properties, such as controlling the "wh-" word or the presence of prepositional phrases.
Limitations and Future Work
Despite its success, the model faces challenges with information loss; compressing a sentence into a fixed-length vector occasionally loses specific details like numbers or rare entities. Additionally, while the syntactic space is discrete, it lacks an inherent ordering, making smooth transitions between styles difficult. Future directions include exploring more powerful code prediction models and reducing the reliance on pre-defined paraphrase clusters during training.
Summary updated