Introduction
Statistical natural language parsers are critical components in modern NLP pipelines, yet their performance is highly sensitive to domain shifts. A parser optimized for financial news often struggles when applied to diverse genres like fiction or medical literature. Traditionally, assessing parser quality on a new domain requires expensive, expert-led manual annotation to create a "gold standard." This paper presents a fully automated alternative: a regression-based approach to predict parser accuracy (F-score) on unannotated text.
Methodology
The authors treat accuracy prediction as a regression problem, employing SVM-Regression to map various features to predicted F-scores. The feature space is divided into three primary categories:
- Text-based Features: These include surface-level metrics such as sentence length, the frequency of "unknown" words (out-of-vocabulary relative to training data), and language model perplexity.
- Parser-based Features: Structural characteristics of the generated parse trees, such as the syntactic category of the root node and punctuation density.
- Reference Parser Agreement: The most innovative feature involves comparing the output of the primary parser with a second, independent "reference" parser. High structural agreement between two distinct models serves as a robust proxy for correctness.
To ensure predictions are well-calibrated across domains, the authors utilize Shifting and Skewing techniques, aligning the predicted distribution with the expected performance range of the target domain.
Experimental Setup
The system was evaluated using several standard benchmarks, including the Wall Street Journal (WSJ) corpus for in-domain testing and the Brown Corpus for out-of-domain evaluation. As shown in the baseline data, parser accuracy drops significantly when moving from financial news to general English text.
Key Results and Insights
The study reveals that while predicting the accuracy of individual sentences is challenging ($r = 0.42$), aggregating sentences into "chunks" significantly improves reliability. For 500-sentence chunks, the correlation between predicted and actual accuracy reaches 0.79, demonstrating that errors average out over larger samples.
In domain adaptation tests on the Brown corpus, the "Adjusted Predictor" estimated an F-score of 86.96, remarkably close to the actual measured accuracy of 86.34. This represents a significant improvement over the naive assumption that the parser would maintain its WSJ-level performance (91.13%). Furthermore, the system proved highly effective at ranking different parser settings on completely new data (Xinhua), even when absolute score predictions were slightly off.
Conclusion
This automated approach provides a viable path for monitoring parser performance in production environments without human intervention. While lexical features sometimes overfit specific domains, the use of reference parser agreement offers a stable and generalizable signal. Future work will explore using a consensus of multiple reference parsers and integrating these predictions into downstream tasks like machine translation to dynamically select the best parsing strategy.
Summary updated