PaLM: A Hybrid Parser and Language Model

Hao Peng, Roy Schwartz, Noah A. Smith

Abstract

We present PaLM, a hybrid parser and neural language model. Building on an RNN language model, PaLM adds an attention layer over text spans in the left context. An unsupervised constituency parser can be derived from its attention weights, using a greedy decoding algorithm. We evaluate PaLM on language modeling, and empirically show that it outperforms strong baselines. If syntactic annotations are available, the attention component can be trained in a supervised manner, providing syntactically-informed representations of the context, and further improving language modeling performance.

Introduction

Standard neural language models (LMs) typically process text as a flat sequence of tokens. However, human language is inherently hierarchical, with words grouping into phrases and clauses. While incorporating this structural bias is known to improve performance on various NLP tasks, doing so often incurs significant computational costs. Previous approaches required complex algorithms to marginalize over all possible tree structures or utilized specialized architectures that are difficult to scale.

PaLM (Parser and Language Model) addresses this by introducing a "lightweight" method to inject structural awareness into an LM without the heavy overhead of traditional grammar-based models. The core insight is that attention weights in a span-based model can functionally serve as a constituency parser.

Methods: Span Attention and Parsing

The defining innovation of PaLM is Span Attention. Rather than attending to individual preceding tokens, the model attends to sequences of words, or spans.

Efficient Span Representations

To manage the $O(n^2)$ cost of calculating vectors for every possible span, PaLM utilizes Rational RNNs (RRNNs). Due to their linear recurrence, the representation of any span $[i, j]$ can be computed efficiently via element-wise operations on gate products.

The Attention Layer and Parser

The model calculates attention scores for preceding spans, and the resulting context vector—a weighted sum of these representations—is used to predict the next word. Because these weights indicate span "importance," they can be used to induce a tree structure. PaLM employs a greedy top-down algorithm that identifies the highest-scoring span, splits the sentence at that point, and recurses.

PaLM architecture: The LM attends over spans to predict the next token, while the parser recursively splits spans based on attention weights.

Training Paradigms

PaLM can be trained in two modes:

  • PaLM-U (Unsupervised): Trained on raw text, where the parser is induced solely from the language modeling objective.
  • PaLM-S (Supervised): The attention mechanism is explicitly guided by gold-standard syntactic trees.

Experimental Results

The authors evaluated PaLM on standard benchmarks including the Penn Treebank (PTB) and WikiText-2 (WT2).

Language Modeling Performance

PaLM-S demonstrated superior performance, achieving a test perplexity of 55.5 on PTB, significantly outperforming the AWD-LSTM baseline of 57.3. On WikiText-2, it improved perplexity from 65.8 to 63.2. These results suggest that even without gold trees, the capability to attend to spans improves predictive accuracy.

Language modeling performance on the Penn Treebank (PTB) dataset. PaLM-S achieves the lowest perplexity.

Unsupervised Parsing and Ablations

In the unsupervised setting (PaLM-U), the model achieved an unlabeled $F_1$ score of 42.0. While lower than specialized parsers like DIORA (60.6), it outperformed a naive right-branching baseline (40.7), indicating that the LM objective captures meaningful structural signals. Ablation studies showed that forcing a right-branching structure (PaLM-RB) yielded no improvement, confirming that PaLM's specific span selections are critical to its success.

Discussion and Future Work

While PaLM effectively integrates syntax and language modeling, it exhibits a right-branching bias in the unsupervised setting, producing more right-skewed trees than found in gold-standard data. Additionally, for computational efficiency, the maximum span length was limited to 20 tokens, which may hinder the capture of very long-distance dependencies. Future research could explore applying these span-based structural biases to Transformer architectures, which currently dominate the field.

Summary updated

Abstract

We present PaLM, a hybrid parser and neural language model. Building on an RNN language model, PaLM adds an attention layer over text spans in the left context. An unsupervised constituency parser can be derived from its attention weights, using a greedy decoding algorithm. We evaluate PaLM on language modeling, and empirically show that it outperforms strong baselines. If syntactic annotations are available, the attention component can be trained in a supervised manner, providing syntactically-informed representations of the context, and further improving language modeling performance.
BibTeX
@inproceedings{peng2019palm,
  author = {Hao Peng and Roy Schwartz and Noah A. Smith},
  title = {{PaLM: A Hybrid Parser and Language Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=O2Mt6WaXiFG},
}
Figure