Abstract: While the average performance of statistical parsers gradually improves, they still attach to many sentences annotations of rather low quality. The number of such sentences grows when the training and test data are taken from different domains, which is the case for major web applications such as information retrieval and question answering. In this paper we present a Sample Ensemble Parse Assessment (SEPA) algorithm for detecting parse quality. We use a function of the agreement among several copies of a parser, each of which trained on a different sample from the training data, to assess parse quality. We experimented with both generative and reranking parsers (Collins, Charniak and Johnson respectively). We show superior results over several baselines, both when the training and test data are from the same domain and when they are from different domains. For a test setting used by previous work, we show an error reduction of 31% as opposed to their 20%.
1. Introduction
Statistical parsers are essential components in high-level Natural Language Processing (NLP) tasks such as Information Extraction and Question Answering. Despite steady improvements in average parsing accuracy, these systems frequently produce low-quality parses that can lead to catastrophic failures in downstream applications. This problem is particularly pronounced in domain adaptation scenarios, where models trained on formal text (e.g., the Wall Street Journal) are applied to the diverse and often noisy language found on the web. Identifying these "low-quality" parses is critical for system reliability, yet standard metrics like internal model probability or sentence length are often poor predictors of actual parse accuracy.
2. The SEPA Algorithm
We introduce the Sample Ensemble Parse Assessment (SEPA) algorithm, a model-agnostic method for identifying high-quality parses without relying on external data sources. SEPA is based on the principle of stability: if a parser is confident in the structure of a sentence, it should produce consistent results even when its training data is slightly perturbed.
The algorithm proceeds in four stages:
- Sampling: Create $N$ random samples from the original training corpus.
- Ensemble Training: Train $N$ independent instances of the same parsing model, each on a different sample.
- Agreement Scoring: For a new sentence, generate $N$ parses and calculate the Mean F-score (MF), representing the average agreement between all pairs of parses in the ensemble.
- Selection: Sentences with an MF score above a specific threshold are selected as high-quality.
As illustrated in Figure 2, the number of models $N$ allows for a tunable trade-off between the precision of the filter and the number of sentences retained (recall).
3. Experimental Evaluation
We evaluated SEPA using the WSJ Penn Treebank (Sections 02-21) for training. Testing was conducted on both in-domain data (WSJ Section 23) and out-of-domain data (the Brown Corpus). We applied SEPA to both the Collins generative parser and the Charniak reranking parser.
4. Results and Discussion
SEPA consistently outperformed the Confidence Baseline (CB), which uses the parser's own probability score, and the Minimum Length (ML) baseline.
Key results include:
- Error Reduction: On a standard WSJ benchmark, SEPA achieved a 31% error reduction, significantly outperforming the 20% reduction reported for the previous state-of-the-art WOODWARD algorithm.
- Domain Robustness: In domain adaptation tasks, SEPA achieved error reductions between 44% and 66% at high agreement thresholds, demonstrating its utility for web-scale applications.
- Intrinsic Assessment: Unlike previous methods that required external semantic checks via web queries, SEPA is entirely intrinsic and does not require internet access or external knowledge bases.
5. Conclusion
SEPA provides a robust, model-agnostic framework for assessing parse quality through ensemble agreement. While the initial training of $N$ models is computationally expensive, the inference process is highly parallelizable. Future research will explore integrating the SEPA agreement signal directly into the parsing and reranking process to improve initial parse selection.
Summary updated