An Ensemble Method for Selection of High Quality Parses

Roi Reichart, Ari RappoportACL 2007 · OpenReview

Abstract: While the average performance of statistical parsers gradually improves, they still attach to many sentences annotations of rather low quality. The number of such sentences grows when the training and test data are taken from different domains, which is the case for major web applications such as information retrieval and question answering. In this paper we present a Sample Ensemble Parse Assessment (SEPA) algorithm for detecting parse quality. We use a function of the agreement among several copies of a parser, each of which trained on a different sample from the training data, to assess parse quality. We experimented with both generative and reranking parsers (Collins, Charniak and Johnson respectively). We show superior results over several baselines, both when the training and test data are from the same domain and when they are from different domains. For a test setting used by previous work, we show an error reduction of 31% as opposed to their 20%.

1. Introduction

Statistical parsers are essential components in high-level Natural Language Processing (NLP) tasks such as Information Extraction and Question Answering. Despite steady improvements in average parsing accuracy, these systems frequently produce low-quality parses that can lead to catastrophic failures in downstream applications. This problem is particularly pronounced in domain adaptation scenarios, where models trained on formal text (e.g., the Wall Street Journal) are applied to the diverse and often noisy language found on the web. Identifying these "low-quality" parses is critical for system reliability, yet standard metrics like internal model probability or sentence length are often poor predictors of actual parse accuracy.

2. The SEPA Algorithm

We introduce the Sample Ensemble Parse Assessment (SEPA) algorithm, a model-agnostic method for identifying high-quality parses without relying on external data sources. SEPA is based on the principle of stability: if a parser is confident in the structure of a sentence, it should produce consistent results even when its training data is slightly perturbed.

The algorithm proceeds in four stages:

  1. Sampling: Create $N$ random samples from the original training corpus.
  2. Ensemble Training: Train $N$ independent instances of the same parsing model, each on a different sample.
  3. Agreement Scoring: For a new sentence, generate $N$ parses and calculate the Mean F-score (MF), representing the average agreement between all pairs of parses in the ensemble.
  4. Selection: Sentences with an MF score above a specific threshold are selected as high-quality.
Trade-off between filter recall and precision as the number of ensemble models (N) increases.

As illustrated in Figure 2, the number of models $N$ allows for a tunable trade-off between the precision of the filter and the number of sentences retained (recall).

3. Experimental Evaluation

We evaluated SEPA using the WSJ Penn Treebank (Sections 02-21) for training. Testing was conducted on both in-domain data (WSJ Section 23) and out-of-domain data (the Brown Corpus). We applied SEPA to both the Collins generative parser and the Charniak reranking parser.

4. Results and Discussion

SEPA consistently outperformed the Confidence Baseline (CB), which uses the parser's own probability score, and the Minimum Length (ML) baseline.

Performance of SEPA vs. baselines across different parsers and domains. The top row shows in-domain results, while the bottom row shows domain adaptation results. The rightmost column tracks the number of sentences remaining at each agreement threshold.

Key results include:

  • Error Reduction: On a standard WSJ benchmark, SEPA achieved a 31% error reduction, significantly outperforming the 20% reduction reported for the previous state-of-the-art WOODWARD algorithm.
  • Domain Robustness: In domain adaptation tasks, SEPA achieved error reductions between 44% and 66% at high agreement thresholds, demonstrating its utility for web-scale applications.
  • Intrinsic Assessment: Unlike previous methods that required external semantic checks via web queries, SEPA is entirely intrinsic and does not require internet access or external knowledge bases.

5. Conclusion

SEPA provides a robust, model-agnostic framework for assessing parse quality through ensemble agreement. While the initial training of $N$ models is computationally expensive, the inference process is highly parallelizable. Future research will explore integrating the SEPA agreement signal directly into the parsing and reranking process to improve initial parse selection.

Summary updated

Abstract

While the average performance of statistical parsers gradually improves, they still attach to many sentences annotations of rather low quality. The number of such sentences grows when the training and test data are taken from different domains, which is the case for major web applications such as information retrieval and question answering. In this paper we present a Sample Ensemble Parse Assessment (SEPA) algorithm for detecting parse quality. We use a function of the agreement among several copies of a parser, each of which trained on a different sample from the training data, to assess parse quality. We experimented with both generative and reranking parsers (Collins, Charniak and Johnson respectively). We show superior results over several baselines, both when the training and test data are from the same domain and when they are from different domains. For a test setting used by previous work, we show an error reduction of 31% as opposed to their 20%.
BibTeX
@inproceedings{reichart2007ensemble,
  author = {Roi Reichart and Ari Rappoport},
  title = {{An Ensemble Method for Selection of High Quality Parses}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2007},
  url = {https://openreview.net/forum?id=r1bAkaxOWB},
}
Figure