Abstract
Traditional language models often overlook the hierarchical structure of natural language, treating text as flat sequences. This paper introduces the Neural Variational Language Model (NVLM), a framework that bridges sequential modeling and structural syntax. By treating grammar as a shared resource, NVLM can learn structural rules from small annotated datasets and transfer this knowledge to large, unannotated corpora. Our results demonstrate that NVLM achieves faster convergence and significantly lower perplexity than standard sequential models, even with limited training data.
Introduction
Standard language models, such as LSTMs or GRUs, typically process language as a linear sequence of tokens. While effective, these models ignore the inherent hierarchical structure—the grammar—of natural language. Structure-aware models like Recurrent Neural Network Grammars (RNNGs) have shown promise, but they are often constrained by the need for expensive, human-annotated treebanks like the Penn Treebank (PTB).
This research addresses two primary challenges: the scarcity of annotated data and the performance drop caused by domain shifts. The authors propose a method to leverage grammatical knowledge from a small annotated corpus to enhance language modeling on a completely different, unannotated target corpus.
The Neural Variational Language Model (NVLM)
The NVLM architecture consists of two main probabilistic components: a constituency parser and a joint generative model.
- Constituency Parser ($P_{\theta_1}$): An encoder-decoder model with attention that predicts a linearized parse tree for a given sentence.
- Joint Generative Model ($P_{\theta_2}$): An RNN that models the "mixed tree"—a sequence combining the syntax tree structure with the actual words of the sentence.
Training Schemes
NVLM supports multiple training paradigms:
- Supervised: Standard training using sentences and their corresponding parse trees.
- Distant-supervised: A pre-trained parser provides "silver" labels for an unannotated corpus.
- Semi-supervised (Joint Training): Utilizing Variational EM and Policy Gradient (REINFORCE), the model fine-tunes both the parser and the generative model on unannotated data. The parser acts as a latent variable, optimizing the Evidence Lower Bound (ELBO).
Experimental Evaluation
The model was evaluated on the Penn Treebank (PTB) for supervised learning and the One Billion Word Benchmark (OBWB) for testing grammar transfer to unannotated data.
Key Results and Data Efficiency
NVLM achieved state-of-the-art results on PTB with a test perplexity of 91.6. On the OBWB dataset, NVLM consistently outperformed standard GRU models. Notably, the "warmed-up" NVLM (pre-trained on PTB) showed remarkable data efficiency and faster convergence.
As shown in the figures above, even when trained on only 20% of the target corpus, the warmed-up NVLM achieved perplexity levels comparable to a standard GRU trained on the full dataset. This suggests that structural knowledge provides a superior inductive bias for language learning.
Discussion and Future Work
The core insight of this work is that while vocabulary and topics vary across domains, the underlying grammar remains relatively stable. By linearizing trees into "mixed" sequences, the authors transform a structural problem into a sequence modeling task manageable by modern RNNs.
However, the model's reliance on Importance Sampling for evaluation introduces computational overhead. Future research could explore integrating these structural priors into Transformer architectures, which currently dominate the field but often lack explicit hierarchical modeling. Additionally, addressing fixed vocabulary constraints would improve the model's robustness across diverse domains.
Summary updated