Language Modeling with Shared Grammar

Yuyu Zhang, Le SongACL 2019 · DOI

Abstract

Traditional language models often overlook the hierarchical structure of natural language, treating text as flat sequences. This paper introduces the Neural Variational Language Model (NVLM), a framework that bridges sequential modeling and structural syntax. By treating grammar as a shared resource, NVLM can learn structural rules from small annotated datasets and transfer this knowledge to large, unannotated corpora. Our results demonstrate that NVLM achieves faster convergence and significantly lower perplexity than standard sequential models, even with limited training data.

Introduction

Standard language models, such as LSTMs or GRUs, typically process language as a linear sequence of tokens. While effective, these models ignore the inherent hierarchical structure—the grammar—of natural language. Structure-aware models like Recurrent Neural Network Grammars (RNNGs) have shown promise, but they are often constrained by the need for expensive, human-annotated treebanks like the Penn Treebank (PTB).

This research addresses two primary challenges: the scarcity of annotated data and the performance drop caused by domain shifts. The authors propose a method to leverage grammatical knowledge from a small annotated corpus to enhance language modeling on a completely different, unannotated target corpus.

The Neural Variational Language Model (NVLM)

The NVLM architecture consists of two main probabilistic components: a constituency parser and a joint generative model.

NVLM Architecture: The model uses an encoder-decoder parser to generate parse trees, which are then merged with the sentence to form a mixed tree for the joint generative model.
  1. Constituency Parser ($P_{\theta_1}$): An encoder-decoder model with attention that predicts a linearized parse tree for a given sentence.
  2. Joint Generative Model ($P_{\theta_2}$): An RNN that models the "mixed tree"—a sequence combining the syntax tree structure with the actual words of the sentence.

Training Schemes

NVLM supports multiple training paradigms:

  • Supervised: Standard training using sentences and their corresponding parse trees.
  • Distant-supervised: A pre-trained parser provides "silver" labels for an unannotated corpus.
  • Semi-supervised (Joint Training): Utilizing Variational EM and Policy Gradient (REINFORCE), the model fine-tunes both the parser and the generative model on unannotated data. The parser acts as a latent variable, optimizing the Evidence Lower Bound (ELBO).

Experimental Evaluation

The model was evaluated on the Penn Treebank (PTB) for supervised learning and the One Billion Word Benchmark (OBWB) for testing grammar transfer to unannotated data.

Key Results and Data Efficiency

NVLM achieved state-of-the-art results on PTB with a test perplexity of 91.6. On the OBWB dataset, NVLM consistently outperformed standard GRU models. Notably, the "warmed-up" NVLM (pre-trained on PTB) showed remarkable data efficiency and faster convergence.

Test perplexity comparison on 100% and 20% training data. NVLM (warmed) consistently achieves lower perplexity and faster convergence than baseline models.

As shown in the figures above, even when trained on only 20% of the target corpus, the warmed-up NVLM achieved perplexity levels comparable to a standard GRU trained on the full dataset. This suggests that structural knowledge provides a superior inductive bias for language learning.

Discussion and Future Work

The core insight of this work is that while vocabulary and topics vary across domains, the underlying grammar remains relatively stable. By linearizing trees into "mixed" sequences, the authors transform a structural problem into a sequence modeling task manageable by modern RNNs.

However, the model's reliance on Importance Sampling for evaluation introduces computational overhead. Future research could explore integrating these structural priors into Transformer architectures, which currently dominate the field but often lack explicit hierarchical modeling. Additionally, addressing fixed vocabulary constraints would improve the model's robustness across diverse domains.

Summary updated

BibTeX
@inproceedings{zhang2019language,
  author = {Yuyu Zhang and Le Song},
  title = {{Language Modeling with Shared Grammar}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1437},
}
Figure