Simple Semi-Supervised Learning for Prepositional Phrase Attachment

Gregory F. Coppola, Alexandra Birch, Tejaswini Deoskar, Mark SteedmanIWPT 2011 · OpenReview

Abstract

Prepositional phrase (PP) attachment is a fundamental challenge in natural language processing, requiring nuanced lexical knowledge to resolve structural ambiguities. This paper presents a semi-supervised approach that combines a discriminative model trained on limited labeled data with a generative model that learns from millions of unlabeled sentences via Expectation Maximization (EM). By introducing a collective dependent generation strategy and utilizing a logarithmic opinion pool for model combination, the proposed system significantly outperforms state-of-the-art parsers, demonstrating particular robustness on out-of-domain text.

Introduction

In natural language processing, prepositional phrase attachment is a notorious source of ambiguity. Consider the sentence: "He ate a salad with a fork." Determining whether "with a fork" modifies the verb "ate" (the instrument) or the noun "salad" (the accompaniment) requires more than just structural rules; it requires deep lexical knowledge of how specific verbs, nouns, and prepositions interact.

While humans use world knowledge to disambiguate these structures, computers struggle because labeled data—sentences hand-parsed by experts—is expensive and scarce. Conversely, unlabeled text is abundant. This paper addresses the challenge of effectively leveraging this "cheap" unlabeled data to improve attachment accuracy, especially when encountering vocabulary not present in small labeled training sets.

Methodology

The authors propose a "product model" that integrates two distinct machine learning paradigms:

  1. The Discriminative Model: This model is trained on the Wall Street Journal (WSJ) portion of the Penn Treebank using a structured perceptron. It evaluates features such as the head word, the preposition, and the noun within the PP. The authors evaluated both "first-order" (head-preposition pairs) and "second-order" (including the internal noun) variants.
  2. The Generative Model: This model learns from unlabeled data using the Expectation Maximization (EM) algorithm. A key contribution is the Collective Dependent Generation strategy. Unlike standard models that predict modifiers independently, this approach generates the entire set of modifiers for a head word simultaneously, capturing the intuition that the presence of one modifier (like a direct object) influences the likelihood of another (like a PP).
  3. Model Combination (Logarithmic Opinion Pool): To synthesize these approaches, the models are combined using a weighted product. This allows the discriminative model to provide high-precision anchors from labeled data while the generative model fills lexical gaps using patterns discovered in millions of unlabeled sentences.

Experimental Setup

  • Labeled Data: The Penn Treebank (WSJ) was used for in-domain training and testing, while the Brown Corpus served as an out-of-domain test set to evaluate generalization.
  • Unlabeled Data: Over 8 million examples were extracted from the New York Times portion of the GigaWord corpus. These examples were automatically parsed to identify potential attachment ambiguities for the model to resolve during the EM process.

Results and Analysis

The proposed discriminative models showed immediate gains over standard parsers. As shown in Figure 1, the second-order discriminative model ($p_{D_2}$) consistently outperformed the baseline Berkeley parser on both WSJ and Brown test sets.

Performance of discriminative classifiers against the baseline Berkeley parser.

One of the most significant findings was the advantage of the "Collective" generation strategy over the standard "Independent" approach. As illustrated in Figure 2, the collective model consistently achieved higher accuracy across EM iterations, particularly on the out-of-domain Brown corpus.

Comparison of Independent vs. Collective generation strategies across EM iterations.

The final combined model (LOP) achieved 86.9% accuracy on the WSJ test set and 86.2% on the Brown corpus, representing a substantial improvement over the Berkeley parser's 83.0% and 82.7%, respectively. Notably, the semi-supervised model parsed out-of-domain data nearly as accurately as in-domain data, proving its robustness.

Discussion and Future Work

The authors observed a phenomenon of "EM drift," where performance on labeled data occasionally decreased as EM iterations progressed. This suggests the model may begin to favor broad patterns in unlabeled data over the high-quality constraints of the labeled set. Future research could explore techniques like Posterior Regularization to maintain structural constraints during training. Additionally, incorporating semantic resources like WordNet or word embeddings could help the model group similar words (e.g., "spoon" and "fork") to further mitigate data sparsity.

Conclusion

By combining discriminative precision with generative scale and a novel collective modeling strategy, this work provides a simple yet powerful framework for semi-supervised PP attachment. The results highlight that deep lexical knowledge, harvested from unlabeled text, is essential for breaking the performance ceiling of traditional structural parsers.

Summary updated

BibTeX
@inproceedings{coppola2011simple,
  author = {Gregory F. Coppola and Alexandra Birch and Tejaswini Deoskar and Mark Steedman},
  title = {{Simple Semi-Supervised Learning for Prepositional Phrase Attachment}},
  booktitle = {IWPT},
  year = {2011},
  url = {https://openreview.net/forum?id=XO7lxukG5n},
}
Figure