Syntax-Infused Variational Autoencoder for Text Generation

Xinyuan Zhang, Yi Yang, Siyang Yuan, Dinghan Shen, Lawrence Carin

Abstract: Xinyuan Zhang, Yi Yang, Siyang Yuan, Dinghan Shen, Lawrence Carin. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL).

2019.

Introduction

Variational Autoencoders (VAEs) have become a staple in generative text modeling. However, they frequently struggle to produce grammatically coherent sentences, often resulting in "mushy" outputs that fail to respect complex linguistic constraints like long-distance subject-verb agreement. This paper argues that the primary bottleneck is the conflation of semantic content and syntactic structure into a single latent vector. To address this, the authors introduce the Syntax-Infused Variational Autoencoder (SIVAE), which explicitly decouples these two components into distinct latent spaces.

Methodology: Decoupling Content and Form

The core innovation of SIVAE is its dual-latent architecture. By providing the model with explicit syntactic information—specifically constituency parse trees—the authors "infuse" the model with structural awareness.

The SIVAE architecture utilizes separate encoders and decoders for sentences and their corresponding syntactic trees, mapping them into distinct latent spaces $z_x$ and $z_y$.

The authors propose two variants of the model:

  1. SIVAE-c (Conditional): This model assumes a hierarchical relationship where syntax is conditioned on semantics ($z_y$ depends on $z_x$). This mirrors the human process of selecting a topic before deciding on a grammatical structure.
  2. SIVAE-i (Independent): This version treats semantics and syntax as independent variables. This independence enables "plug-and-play" generation, allowing the model to combine the meaning of one sentence with the syntactic template of another for unsupervised paraphrasing.

Technical Implementation

To process hierarchical trees with sequence-based LSTMs, SIVAE employs Tree Linearization, converting bracketed parse structures into tag sequences. The model's objective function, the Evidence Lower Bound (ELBO), was redesigned for joint optimization across the two latent spaces. During generation, the sentence decoder is conditioned on both the semantic latent vector and the generated syntactic tree to ensure structural fidelity.

Experimental Setup

The models were evaluated on two datasets:

  • Penn Treebank (PTB): Features high-quality, human-labeled "gold-standard" syntactic trees.
  • Wikipedia-90M (wiki90M): A large-scale dataset using machine-generated trees from a constituency parser, testing the model's robustness to noisy, automated syntax.

Results and Discussion

SIVAE demonstrates significant improvements in both language modeling quality and grammatical precision.

Language modeling results on PTB and wiki90M datasets. SIVAE models consistently achieve lower perplexity (PPL) and negative log-likelihood (NLL) compared to standard VAEs.

Key Findings

  • Improved Language Modeling: SIVAE-i achieved a Perplexity (PPL) of 90 on PTB, a substantial improvement over the standard VAE's PPL of 112.
  • Syntactic Robustness: In targeted evaluations of grammatical rules (e.g., subject-verb agreement), SIVAE-c reached an accuracy of 0.97, outperforming both the baseline VAE (0.94) and even human scores on certain test sets.
  • Unsupervised Paraphrasing: The decoupling in SIVAE-i allows for effective paraphrasing without parallel corpora. Human judges rated SIVAE's paraphrases as more readable and relevant than those from standard VAEs.

Conclusion and Future Directions

By separating "what" is said from "how" it is structured, SIVAE reduces the representational burden on the model, leading to more coherent text generation. While the current model uses linearized trees, future iterations could explore Graph Neural Networks (GNNs) to better capture hierarchical relationships. Additionally, addressing "template mismatch"—where a semantic meaning is forced into an incompatible syntactic structure—remains an open challenge for improving the reliability of controlled text generation.

Summary updated

BibTeX
@inproceedings{zhang2019syntax,
  author = {Xinyuan Zhang and Yi Yang and Siyang Yuan and Dinghan Shen and Lawrence Carin},
  title = {{Syntax-Infused Variational Autoencoder for Text Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=x5Qnq-w45O5},
}
Figure