Factorising Meaning and Form for Intent-Preserving Paraphrasing

Tom Hosking, Mirella Lapata

Abstract

Paraphrasing requires balancing semantic consistency with syntactic diversity. We present SEPARATOR, a model that generates diverse paraphrases by explicitly disentangling a sentence's meaning from its syntactic form. By using a dual-bottleneck architecture—continuous for meaning and discrete for syntax—the model can "re-skin" the intent of a question into entirely new structures without needing external reference templates at test time.

Introduction

Effective paraphrasing involves transforming the surface form of a sentence while preserving its underlying semantic intent. Traditional models often struggle with this balance, either producing trivial changes (parroting) or requiring human-provided exemplars to guide syntactic variation. We address this by introducing SEPARATOR, a Transformer-based encoder-decoder model designed to factorise latent representations into distinct semantic and syntactic components.

Methodology: The SEPARATOR Architecture

The core innovation of SEPARATOR is its dual-bottleneck latent space. Unlike standard Variational Autoencoders (VAEs) that use a single latent vector, our model splits the representation into two distinct paths:

  1. Semantic Encoding ($z_{sem}$): A continuous space modeled via a VAE. This captures the fluid, high-dimensional nature of meaning.
  2. Syntactic Encoding ($z_{syn}$): A discrete space modeled via a Vector-Quantized Variational Autoencoder (VQ-VAE). This captures the categorical nature of sentence structures (e.g., "How much does X weigh?" vs. "What is the weight of X?").
The SEPARATOR architecture featuring dual continuous and discrete bottlenecks.

To force the model to separate these features, it is trained on triples $(X_{sem}, X_{syn}, Y)$, learning to reconstruct a target $Y$ using the semantic content of $X_{sem}$ and the syntactic structure of $X_{syn}$. At test time, a small classifier predicts valid discrete syntactic codes for a given intent, allowing the model to sample different "syntactic skins" for the same meaning autonomously.

Experiments and Datasets

We evaluated SEPARATOR on two major English question datasets:

  • Paralex: A large, noisy dataset scraped from WikiAnswers containing clusters of questions sharing the same answer.
  • Quora Question Pairs (QQP): A cleaner dataset containing subjective and conversational questions.

To ensure data quality, we applied language detection and spell-checking to the noisy Paralex corpus before training.

Key Results

SEPARATOR demonstrates a superior tradeoff between semantic preservation and syntactic novelty. It achieved the highest iBLEU scores—a metric that penalizes similarity to the input—on both Paralex (14.84) and QQP (5.84), significantly outperforming baselines like DiPS and SOW/REAP.

Human Evaluation

In side-by-side comparisons, human judges assessed the models based on meaning preservation, dissimilarity to the input, and fluency. SEPARATOR was found to be significantly more diverse than standard VAEs while remaining more fluent and meaning-preserving than other diversity-focused models.

Human preference evaluation comparing SEPARATOR against VAE, Latent BoW, and DiPS.

Discussion and Insights

Our analysis reveals that discrete syntax is a natural fit for language structure. While meaning is infinite and nuanced, the number of common ways to structure a question is relatively finite. By analyzing the entropy of the quantizer heads, we found that different heads "learned" specific syntactic properties, such as controlling the "wh-" word or the presence of prepositional phrases.

Limitations and Future Work

Despite its success, the model faces challenges with information loss; compressing a sentence into a fixed-length vector occasionally loses specific details like numbers or rare entities. Additionally, while the syntactic space is discrete, it lacks an inherent ordering, making smooth transitions between styles difficult. Future directions include exploring more powerful code prediction models and reducing the reliance on pre-defined paraphrase clusters during training.

Summary updated

Abstract

We propose a method for generating paraphrases of English questions that retain the original intent but use a different surface form. Our model combines a careful choice of training objective with a principled information bottleneck, to induce a latent encoding space that disentangles meaning and form. We train an encoder-decoder model to reconstruct a question from a paraphrase with the same meaning and an exemplar with the same surface form, leading to separated encoding spaces. We use a Vector-Quantized Variational Autoencoder to represent the surface form as a set of discrete latent variables, allowing us to use a classifier to select a different surface form at test time. Crucially, our method does not require access to an external source of target exemplars. Extensive experiments and a human evaluation show that we are able to generate paraphrases with a better tradeoff between semantic preservation and syntactic novelty compared to previous methods.
BibTeX
@inproceedings{hosking2021factorising,
  author = {Tom Hosking and Mirella Lapata},
  title = {{Factorising Meaning and Form for Intent-Preserving Paraphrasing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.112},
}
Figure