Abstract
We propose a method for generating paraphrases of English questions that retain the original intent but use a different surface form. Our model combines a careful choice of training objective with a principled information bottleneck, to induce a latent encoding space that disentangles meaning and form. We train an encoder-decoder model to reconstruct a question from a paraphrase with the same meaning and an exemplar with the same surface form, leading to separated encoding spaces. We use a Vector-Quantized Variational Autoencoder to represent the surface form as a set of discrete latent variables, allowing us to use a classifier to select a different surface form at test time. Crucially, our method does not require access to an external source of target exemplars. Extensive experiments and a human evaluation show that we are able to generate paraphrases with a better tradeoff between semantic preservation and syntactic novelty compared to previous methods.
Introduction
Paraphrasing is a fundamental task in Natural Language Processing (NLP), essential for data augmentation and improving conversational agents. The objective is to restate an input while preserving its semantic intent. However, existing models often face a "tradeoff of boredom": simple autoencoders tend to replicate the input with minimal changes, while more complex models require external syntactic "exemplars" to guide the generation of novel structures.
The core challenge addressed in this work is how to automatically generate paraphrases that are semantically identical to the original yet syntactically novel, without requiring a predefined target structure at inference time. We introduce SEPARATOR, a model that explicitly disentangles meaning from grammatical form.
Methods: The SEPARATOR Architecture
SEPARATOR is an encoder-decoder model based on the Transformer architecture that utilizes a principled information bottleneck to separate features.
Dual Latent Spaces
Instead of a single representation, the model splits the latent space into two distinct paths:
- Meaning Path ($\mathbf{z}_{sem}$): A continuous space using a Variational Autoencoder (VAE) to capture the "intent" or "gist" of the question.
- Form Path ($\mathbf{z}_{syn}$): A discrete space using a Vector-Quantized VAE (VQ-VAE). This maps the sentence's structure to a "codebook" of discrete syntactic patterns, mirroring the finite nature of common question structures (e.g., "What is X?" vs. "How much is X?").
Training and Inference
To enforce disentanglement, the model is trained on triplets: an input providing the target meaning, an "exemplar" providing the target syntax, and the target output. At test time, where target exemplars are typically unavailable, a small classifier predicts valid syntactic codes from the codebook based on the input's meaning, allowing the model to "hallucinate" a new, valid structure.
Experimental Setup
The model was evaluated on two major question-paraphrase datasets: Paralex (WikiAnswers clusters) and Quora Question Pairs (QQP). Data was rigorously cleaned using language detection and spellchecking to prevent the model from learning to replicate noise or typos.
Results
SEPARATOR was evaluated using iBLEU, a metric that rewards semantic similarity to a reference while penalizing lexical overlap with the original input. SEPARATOR significantly outperformed baselines, achieving an iBLEU of 14.84 on Paralex compared to 8.35 for a standard VAE.
Human Evaluation
In human trials, SEPARATOR demonstrated a superior balance between "dissimilarity" (structural change) and "meaning preservation."
As shown in the evaluation results, while the VAE baseline excels at meaning preservation (often by simply copying the input), it fails significantly at generating dissimilar structures. Conversely, models like DiPS generate high dissimilarity but at a severe cost to meaning and fluency. SEPARATOR provides a more balanced profile, successfully navigating the tradeoff.
Discussion and Conclusion
One of the key insights of this work is that matching the mathematical representation to linguistic reality—nuanced, continuous meaning vs. categorical, discrete syntax—leads to better disentanglement. Verification via t-SNE confirmed that semantic encodings clustered by meaning while syntactic encodings remained independent.
Despite its success, limitations remain. The fixed-length vector compression can lead to numerical errors (e.g., changing "1988" to "1992"), and like many VAEs, the model occasionally suffers from posterior collapse. Future work will focus on creating a more "navigable" syntactic space to allow for finer control over the generated output's formality or style.
Summary updated