Weakly-Supervised Grammar-Informed Bayesian CCG Parser Learning

Dan Garrette, Chris Dyer, Jason Baldridge, Noah A. SmithAAAI 2015 · OpenReview

Abstract

Combinatory Categorial Grammar (CCG) is a lexicalized grammar formalism in which words are associated with categories that, in combination with a small universal set of rules, specify the syntactic configurations in which they may occur. Previous work has shown that learning sequence models for CCG tagging can be improved by using priors that are sensitive to the formal properties of CCG as well as cross-linguistic universals. We extend this approach to the task of learning a full CCG parser from weak supervision. We present a Bayesian formulation for CCG parser induction that assumes only supervision in the form of an incomplete tag dictionary mapping some word types to sets of potential categories. Our approach outperforms a baseline model trained with uniform priors by exploiting universal, intrinsic properties of the CCG formalism to bias the model toward simpler, more cross-linguistically common categories.

Introduction

Building natural language parsers typically requires large-scale "treebanks"—collections of sentences manually annotated with syntactic structures. However, for the vast majority of the world's languages and specialized domains, such resources are unavailable. This research addresses the challenge of weakly-supervised parsing within the Combinatory Categorial Grammar (CCG) framework. CCG is a powerful formalism where lexical categories define how words combine, but its complexity makes it difficult to learn without extensive gold-standard data.

Methodology

The proposed framework treats parsing as a Bayesian inference problem using a Probabilistic Context-Free Grammar (PCFG). The core innovation lies in the use of linguistically-informed priors to guide the learning process:

  1. Category Complexity Prior ($P_G$): This distribution favors simpler grammar categories over complex ones. For instance, a basic noun is assigned a higher prior probability than a complex recursive modifier.
  2. Terminal Production Priors ($P_{em}$): These leverage a partial tag dictionary and raw text to estimate category associations for both known and unknown words.
  3. Inference: The model employs Gibbs sampling, a Markov Chain Monte Carlo (MCMC) approach, to alternate between sampling possible parse trees for unlabeled sentences and updating model parameters.
Example of a CCG parse tree for the sentence "The man walks to work," illustrating the recursive nature of categories.

The parser is constrained by a small set of universal CCG rules and utilizes punctuation as a signal for phrase boundaries to maintain a manageable search space.

Experimental Results

The model was evaluated across three languages—English, Chinese, and Italian—using small training sets (150 to 1,000 sentences) to simulate low-resource conditions. The primary metric was dependency accuracy.

Dependency accuracy results across English, Chinese, and Italian, comparing uniform baselines against grammar-informed priors.

As shown in the results, incorporating grammar-informed priors ($P_G, P_{em}$) consistently outperformed the uniform baseline. The improvement was most significant in Chinese, with a gain of over 6%. Interestingly, while the simplicity prior slightly reduced local supertagging accuracy, it improved global parsing coherence, suggesting that simpler structures provide a better inductive bias for overall tree construction.

Discussion and Conclusion

The results demonstrate that the intrinsic structural properties of CCG can serve as a powerful regularizer for unsupervised learning. By mathematically defining category complexity, the model avoids overfitting to rare, complex edge cases. This approach is particularly robust when supervision is noisy, such as when using unpruned tag dictionaries.

Future work may focus on inducing the tag dictionary itself from scratch or incorporating more complex CCG rules like composition. Additionally, scaling the computationally intensive Gibbs sampling process to larger datasets remains a key challenge for broader application.

Summary updated

Abstract

Combinatory Categorial Grammar (CCG) is a lexicalized grammar formalism in which words are associated with categories that, in combination with a small universal set of rules, specify the syntactic configurations in which they may occur. Previous work has shown that learning sequence models for CCG tagging can be improved by using priors that are sensitive to the formal properties of CCG as well as cross-linguistic universals. We extend this approach to the task of learning a full CCG parser from weak supervision. We present a Bayesian formulation for CCG parser induction that assumes only supervision in the form of an incomplete tag dictionary mapping some word types to sets of potential categories. Our approach outperforms a baseline model trained with uniform priors by exploiting universal, intrinsic properties of the CCG formalism to bias the model toward simpler, more cross-linguistically common categories.
BibTeX
@inproceedings{garrette2015weakly,
  author = {Dan Garrette and Chris Dyer and Jason Baldridge and Noah A. Smith},
  title = {{Weakly-Supervised Grammar-Informed Bayesian CCG Parser Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  url = {https://openreview.net/forum?id=BkWfr1-OZr},
}
Figure