Syntax-Infused Variational Autoencoder for Text Generation

Xinyuan Zhang, Yi Yang, Siyang Yuan, Dinghan Shen, Lawrence Carin

Abstract

We present a syntax-infused variational autoencoder (SIVAE), that integrates sentences with their syntactic trees to improve the grammar of generated sentences. Distinct from existing VAE-based text generative models, SIVAE contains two separate latent spaces, for sentences and syntactic trees. The evidence lower bound objective is redesigned correspondingly, by optimizing a joint distribution that accommodates two encoders and two decoders. SIVAE works with long short-term memory architectures to simultaneously generate sentences and syntactic trees. Two versions of SIVAE are proposed: one captures the dependencies between the latent variables through a conditional prior network, and the other treats the latent variables independently such that syntactically-controlled sentence generation can be performed. Experimental results demonstrate the generative superiority of SIVAE on both reconstruction and targeted syntactic evaluations. Finally, we show that the proposed models can be used for unsupervised paraphrasing given different syntactic tree templates.

Introduction

Deep generative models, particularly Variational Autoencoders (VAEs), have significantly advanced the field of text generation. However, these models frequently encounter difficulties in maintaining grammatical consistency. Standard VAEs attempt to compress both semantic meaning and structural information into a single latent vector. This often results in the loss of fine-grained structural details, leading to generated text that, while semantically coherent, may suffer from grammatical errors such as subject-verb agreement failures.

To address this, we propose the Syntax-Infused Variational Autoencoder (SIVAE). Rather than relying on the model to implicitly learn grammar, SIVAE explicitly incorporates syntactic knowledge—specifically constituency trees—into the generative process.

Methodology

The fundamental innovation of SIVAE is the decoupling of latent representations. Instead of a single latent variable, the model utilizes two distinct variables: $z_x$ for sentence content and $z_y$ for syntactic structure.

The SIVAE architecture featuring separate latent spaces for sentence content ($z_x$) and syntactic structure ($z_y$).

We introduce two variants of the model:

  1. SIVAE-c (Conditional): This variant models syntax as dependent on semantics via a conditional prior network. This approach reflects the human cognitive process of selecting a topic before determining the phrasing.
  2. SIVAE-i (Independent): This variant treats the latent spaces as independent, enabling controlled generation. Users can combine the semantic content of one sentence with the syntactic "shell" of another.

To process tree structures within standard LSTM architectures, we "linearize" the trees into sequences of brackets and tags. The Evidence Lower Bound (ELBO) objective is redesigned to optimize the joint distribution of both text and syntax.

Experimental Setup

SIVAE was evaluated using two datasets:

  • Penn Treebank (PTB): A benchmark with ~40,000 sentences and human-labeled "gold-standard" syntactic trees.
  • Wikipedia-90M (wiki90M): A larger dataset (~70,000 sentences) using "silver-standard" trees generated by a state-of-the-art parser to test model robustness with machine-generated labels.

Results and Discussion

SIVAE demonstrates significant improvements in both reconstruction quality and grammatical accuracy. As shown in the performance metrics, SIVAE-i achieved a Perplexity (PPL) of 90.1 on the PTB dataset, notably outperforming standard VAE and LSTM-LM baselines.

Performance comparison of SIVAE variants against baseline models on PTB and wiki90M datasets, showing improvements in Perplexity (PPL) and Negative Log-Likelihood (NLL).

Key findings include:

  • Mitigation of Posterior Collapse: By requiring the reconstruction of complex syntactic trees, SIVAE ensures that latent spaces remain informative, preventing the common VAE issue where the model ignores the latent vector.
  • Syntactic Scaffolding: Providing an explicit syntactic structure assists the decoder in managing long-distance dependencies, resulting in higher accuracy in targeted syntactic evaluations (e.g., 0.97 accuracy in subject-verb agreement for SIVAE-c).
  • Unsupervised Paraphrasing: The separation of meaning and structure allows SIVAE-i to perform paraphrasing without parallel training data by swapping syntactic templates.

Conclusion

SIVAE offers a robust framework for syntax-aware text generation. While the model faces challenges such as computational overhead from long linearized trees and potential template mismatches, its ability to decouple semantics from syntax provides a powerful tool for controlled generation and unsupervised paraphrasing. Future work will explore extending this framework to tasks like low-resource machine translation and style transfer.

Summary updated

Abstract

We present a syntax-infused variational autoencoder (SIVAE), that integrates sentences with their syntactic trees to improve the grammar of generated sentences. Distinct from existing VAE-based text generative models, SIVAE contains two separate latent spaces, for sentences and syntactic trees. The evidence lower bound objective is redesigned correspondingly, by optimizing a joint distribution that accommodates two encoders and two decoders. SIVAE works with long short-term memory architectures to simultaneously generate sentences and syntactic trees. Two versions of SIVAE are proposed: one captures the dependencies between the latent variables through a conditional prior network, and the other treats the latent variables independently such that syntactically-controlled sentence generation can be performed. Experimental results demonstrate the generative superiority of SIVAE on both reconstruction and targeted syntactic evaluations. Finally, we show that the proposed models can be used for unsupervised paraphrasing given different syntactic tree templates.
BibTeX
@inproceedings{zhang2019syntax,
  author = {Xinyuan Zhang and Yi Yang and Siyang Yuan and Dinghan Shen and Lawrence Carin},
  title = {{Syntax-Infused Variational Autoencoder for Text Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=x5Qnq-w45O5},
}
Figure