Effective Self-Training for Parsing

David McClosky, Eugene Charniak, Mark JohnsonNAACL 2006

TL;DR

This paper introduces a simple yet surprisingly effective self-training method for improving syntactic parsers. By combining a generative parser with a discriminative reranker and leveraging a large amount of unlabeled text, they achieve a significant 1.1% absolute improvement in parsing accuracy on the Wall Street Journal dataset, outperforming previous state-of-the-art results.

What Problem Does This Paper Address?

Syntactic parsing, the task of determining the grammatical structure of a sentence, is a fundamental problem in natural language processing. While supervised methods achieve high performance with ample labeled data, obtaining such data is expensive and time-consuming. This paper addresses the challenge of improving parser performance by effectively utilizing readily available unlabeled text, a common scenario in many real-world applications where labeled data is scarce. Traditional self-training approaches have often yielded minimal improvements or even degraded performance in parsing, leading researchers to explore other avenues. This work aims to demonstrate a practical and effective way to harness unlabeled data for parser improvement.

Methods

The core of the proposed method is a two-phase system:

  1. First-Stage 50-Best Parser: This phase uses a lexicalized probabilistic context-free parser (based on Charniak, 2000 and Charniak & Johnson, 2005) that is modified to produce not just the single best parse, but a list of the top 50 most probable parses for each sentence. This "n-best" list provides a richer set of parsing hypotheses.
  2. MaxEnt Reranker: A Maximum Entropy reranker then takes the 50-best parses from the first stage and reorders them to select the single best parse. This reranker uses a large number of features (over 1.3 million) that capture various aspects of the parse trees, including parts-of-speech and surrounding words.

The self-training aspect comes into play by using the output of this two-phase system to augment the training data. Specifically, the reranker-best parses from unlabeled data are treated as "gold standard" and added to the original labeled training data to retrain the first-stage parser. The reranker itself is not retrained with the self-trained data, as initial experiments showed no significant improvement.

Datasets

  • Labeled Data: Penn Treebank (Marcus et al., 1993), specifically Wall Street Journal (WSJ) articles from sections 2-21 for training, section 24 for development, and section 23 for final testing.
  • Unlabeled Data: North American News Text corpus (NANC) (Graff, 1995), comprising approximately 24 million unlabeled sentences from various news sources. For this paper, articles from the LA Times were used.

Key Insights and Ideas

  • Reranker is Key: The success of self-training for parsing in this work is heavily attributed to the discriminative reranker. While the reranker doesn't directly see the unlabeled data, its ability to select more accurate parses from the n-best list is crucial for generating high-quality pseudo-labeled data.
  • Parser-Best vs. Reranker-Best: Simply adding the single best parse from the parser (parser-best) to the training data yielded no improvement, confirming previous findings. However, using the reranker's top choice (reranker-best) led to significant gains. This highlights the importance of the reranker's more sophisticated feature set in identifying better parses.
  • The "Goldilocks Effect": Self-training was most effective for sentences of intermediate length (20-50 words). Very short sentences were already parsed well by the baseline, and very long sentences remained challenging. This suggests that self-training helps most in the "sweet spot" where the parser has some difficulty but is not completely overwhelmed.
  • Conjunctions Improve: Surprisingly, the analysis indicated that self-training improved performance on sentences with conjunctions, which are notoriously difficult for parsers. This is a positive finding, as improvements in this area are hard to come by.
  • Unknown Words and PP Attachment Don't Improve: Contrary to expectations, the error analysis showed no improvement in handling unknown words or prepositional-phrase attachment. The authors hypothesize that the LA Times corpus, used for self-training, might not contain enough of the specific unknown words present in WSJ, or that PP attachment is less sensitive to the language modeling improvements offered by self-training.

Key Results

  • State-of-the-Art Performance: The proposed method achieved an f-score of 92.1% on WSJ section 23, which is a 1.1% absolute improvement (a 12% error reduction) over the previous best result.
  • Parser Improvement: The first-stage parser's one-best f-score improved from 89.7% to 91.0% after incorporating reranker-best sentences from NANC.
  • Reranker Improvement: The reranking parser's f-score improved from 91.0% to 92.1%.
  • Data Efficiency: Significant improvements were observed even with the addition of a relatively small amount of unlabeled data (50,000 sentences), with performance leveling off around 1.75 million sentences.
  • Weighting Matters: Giving the original WSJ training data a higher relative weight (e.g., 5x) in combination with NANC data led to better asymptotic performance.

Limitations and Future Work

  • Unknown Words and PP Attachment: The lack of improvement in handling unknown words and PP attachment is a limitation. The authors suggest verifying their hypothesis about the LA Times corpus not adequately representing WSJ's unknown words.
  • Reranker Retraining: Retraining the reranker with self-trained data did not yield improvements, suggesting that the current reranker architecture might not benefit from this approach.
  • Future Directions:
    • Improved Error Analysis: Further investigation into why certain sentence types (unknown words, PP attachment) don't improve.
  • More Sophisticated Self-Training: Exploring methods like sentence selection (choosing only the most informative self-trained sentences) or learning grammatical generalizations directly.
  • Practicality: Investigating ways to prune the self-trained data to reduce memory requirements, as the current model requires about half a gigabyte of RAM.
  • Genre Differences: Further research into whether the reranking parser can overcome genre differences between training and testing data.

Summary updated

BibTeX
@inproceedings{mcclosky2006effective,
  author = {David McClosky and Eugene Charniak and Mark Johnson},
  title = {{Effective Self-Training for Parsing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure