Abstract
We present a syntax-infused variational autoencoder (SIVAE), that integrates sentences with their syntactic trees to improve the grammar of generated sentences. Distinct from existing VAE-based text generative models, SIVAE contains two separate latent spaces, for sentences and syntactic trees. The evidence lower bound objective is redesigned correspondingly, by optimizing a joint distribution that accommodates two encoders and two decoders. SIVAE works with long short-term memory architectures to simultaneously generate sentences and syntactic trees. Two versions of SIVAE are proposed: one captures the dependencies between the latent variables through a conditional prior network, and the other treats the latent variables independently such that syntactically-controlled sentence generation can be performed. Experimental results demonstrate the generative superiority of SIVAE on both reconstruction and targeted syntactic evaluations. Finally, we show that the proposed models can be used for unsupervised paraphrasing given different syntactic tree templates.
Introduction
Deep generative models, particularly Variational Autoencoders (VAEs), have significantly advanced the field of text generation. However, these models frequently encounter difficulties in maintaining grammatical consistency. Standard VAEs attempt to compress both semantic meaning and structural information into a single latent vector. This often results in the loss of fine-grained structural details, leading to generated text that, while semantically coherent, may suffer from grammatical errors such as subject-verb agreement failures.
To address this, we propose the Syntax-Infused Variational Autoencoder (SIVAE). Rather than relying on the model to implicitly learn grammar, SIVAE explicitly incorporates syntactic knowledge—specifically constituency trees—into the generative process.
Methodology
The fundamental innovation of SIVAE is the decoupling of latent representations. Instead of a single latent variable, the model utilizes two distinct variables: $z_x$ for sentence content and $z_y$ for syntactic structure.
We introduce two variants of the model:
- SIVAE-c (Conditional): This variant models syntax as dependent on semantics via a conditional prior network. This approach reflects the human cognitive process of selecting a topic before determining the phrasing.
- SIVAE-i (Independent): This variant treats the latent spaces as independent, enabling controlled generation. Users can combine the semantic content of one sentence with the syntactic "shell" of another.
To process tree structures within standard LSTM architectures, we "linearize" the trees into sequences of brackets and tags. The Evidence Lower Bound (ELBO) objective is redesigned to optimize the joint distribution of both text and syntax.
Experimental Setup
SIVAE was evaluated using two datasets:
- Penn Treebank (PTB): A benchmark with ~40,000 sentences and human-labeled "gold-standard" syntactic trees.
- Wikipedia-90M (wiki90M): A larger dataset (~70,000 sentences) using "silver-standard" trees generated by a state-of-the-art parser to test model robustness with machine-generated labels.
Results and Discussion
SIVAE demonstrates significant improvements in both reconstruction quality and grammatical accuracy. As shown in the performance metrics, SIVAE-i achieved a Perplexity (PPL) of 90.1 on the PTB dataset, notably outperforming standard VAE and LSTM-LM baselines.
Key findings include:
- Mitigation of Posterior Collapse: By requiring the reconstruction of complex syntactic trees, SIVAE ensures that latent spaces remain informative, preventing the common VAE issue where the model ignores the latent vector.
- Syntactic Scaffolding: Providing an explicit syntactic structure assists the decoder in managing long-distance dependencies, resulting in higher accuracy in targeted syntactic evaluations (e.g., 0.97 accuracy in subject-verb agreement for SIVAE-c).
- Unsupervised Paraphrasing: The separation of meaning and structure allows SIVAE-i to perform paraphrasing without parallel training data by swapping syntactic templates.
Conclusion
SIVAE offers a robust framework for syntax-aware text generation. While the model faces challenges such as computational overhead from long linearized trees and potential template mismatches, its ability to decouple semantics from syntax provides a powerful tool for controlled generation and unsupervised paraphrasing. Future work will explore extending this framework to tasks like low-resource machine translation and style transfer.
Summary updated