Better Informed Training of Latent Syntactic Features

Markus Dreyer, Jason EisnerEMNLP 2006 · OpenReview

Abstract

We study unsupervised methods for learning refinements of the nonterminals in a treebank. Following Matsuzaki et al. (2005) and Prescher (2005), we may for example split NP without supervision into NP[0] and NP[1], which behave differently. We first propose to learn a PCFG that adds such features to nonterminals in such a way that they respect patterns of linguistic feature passing: each node's nonterminal features are either identical to, or independent of, those of its parent. This linguistic constraint reduces runtime and the number of parameters to be learned. However, it did not yield improvements when training on the Penn Treebank. An orthogonal strategy was more successful: to improve the performance of the EM learner by treebank preprocessing and by annealing methods that split nonterminals selectively. Using these methods, we can maintain high parsing accuracy while dramatically reducing the model size.

Introduction

Standard Probabilistic Context-Free Grammars (PCFGs) derived from treebanks often suffer from being too coarse. A single label like "Noun Phrase" (NP) is expected to cover a wide variety of syntactic roles, such as subjects and objects, which exhibit distinct statistical behaviors. While earlier methods relied on manual splitting or lexicalization, recent advances introduced the PCFG-LA (Latent Annotation) model. This model uses the Expectation-Maximization (EM) algorithm to automatically learn these refinements. However, PCFG-LA is computationally demanding—with parameter growth scaling cubically ($O(L^3)$) with the number of latent states—and often fails to incorporate known linguistic principles of feature propagation.

The INHERIT Model

The authors' primary innovation is the INHERIT model, a linguistically motivated constraint designed to streamline the learning of latent features. In a standard unconstrained model, every child node can have a latent feature completely independent of its parent. The INHERIT model, by contrast, restricts a child node to either inherit its parent's feature or select a new one independently.

The INHERIT model constrains children to inherit the latent features of their parent nodes, reducing parameter complexity.

This constraint significantly reduces the parameter complexity for binary rules from $O(L^3)$ to $O(L)$, potentially leading to faster training and more interpretable features. The intuition is that features like number or tense often propagate through the tree from a head to its constituents.

Training Strategies and Selective Splitting

Beyond the INHERIT model, the paper explores strategies to overcome the local maxima issues inherent in EM training:

  1. Annealing-Like Training: The authors tested "Clone All," which doubles states across the board, and "Clone Some," which selectively splits only the most informative nonterminals. The latter uses Jensen-Shannon divergence to identify categories where refinement provides the most statistical gain.
  2. Markovization: Pre-processing the treebank with horizontal and vertical markovization provides a more structured "backbone" for the EM learner, incorporating parent and sibling context into the labels before latent features are even added.

Key Insights and Results

Despite its theoretical elegance, the INHERIT model was a disappointment in terms of raw accuracy. While it successfully identified linguistic features like singular/plural distinctions, it was outperformed by the unconstrained PCFG-LA model. This suggests that latent states in unconstrained models capture complex statistical regularities that do not always align with human linguistic categories.

Linguistic features like number (sg) and tense (pres) passing through a parse tree, which the INHERIT model attempts to capture.

However, the "Clone Some" selective splitting method was highly successful. It achieved an $F_1$ score of 86.17 on the Penn Treebank test set while using 30% fewer parameters than a standard model. This demonstrates that high-accuracy parsing can be achieved more efficiently by focusing refinements on the most ambiguous grammatical categories, such as Noun Phrases, rather than splitting every symbol uniformly.

Conclusion

The study highlights a tension between linguistic intuition and statistical flexibility. While hard-coded inheritance rules proved too rigid, the success of selective splitting and markovization offers a path toward more efficient grammar induction. Future work may involve using log-linear models to implement these linguistic constraints as "soft" weights rather than hard rules, allowing the model to benefit from linguistic structure without sacrificing the flexibility needed to capture complex data patterns.

Summary updated

Abstract

We study unsupervised methods for learning refinements of the nonterminals in a treebank. Following Matsuzaki et al. (2005) and Prescher (2005), we may for example split NP without supervision into NP[0] and NP[1], which behave differently. We first propose to learn a PCFG that adds such features to nonterminals in such a way that they respect patterns of linguistic feature passing: each node's nonterminal features are either identical to, or independent of, those of its parent. This linguistic constraint reduces runtime and the number of parameters to be learned. However, it did not yield improvements when training on the Penn Treebank. An orthogonal strategy was more successful: to improve the performance of the EM learner by treebank preprocessing and by annealing methods that split nonterminals selectively. Using these methods, we can maintain high parsing accuracy while dramatically reducing the model size.
BibTeX
@inproceedings{dreyer2006better,
  author = {Markus Dreyer and Jason Eisner},
  title = {{Better Informed Training of Latent Syntactic Features}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2006},
  url = {https://openreview.net/forum?id=Bk4xSzzuZB},
}
Figure