Evaluating the Effects of Treebank Size in a Practical Application for Parsing

Kenji Sagae, Yusuke Miyao, Rune Saetre, Jun'ichi TsujiiSETQALNLP 2008 · OpenReview

Abstract Kenji Sagae, Yusuke Miyao, Rune Saetre, Jun’ichi Tsujii. Software Engineering, Testing, and Quality Assurance for Natural Language Processing.

2008.

Introduction

In the field of Natural Language Processing (NLP), components such as part-of-speech taggers and syntactic parsers are frequently evaluated in isolation using standardized metrics like F-score or Labeled Attachment Score. However, the extent to which a marginal improvement in these metrics translates to the performance of a practical, end-to-end application remains an open question. This paper bridges the gap by investigating how the size of a training treebank affects a system designed for Protein-Protein Interaction (PPI) extraction from biomedical literature.

The authors address three primary questions: the amount of domain-specific data required for a useful parser, the quantitative relationship between parse accuracy and application performance, and the optimal allocation of annotation resources between syntactic trees and task-specific examples.

Methodology

The researchers developed a pipeline for PPI extraction, a binary classification task identifying interactions between proteins mentioned in text.

The Parsing Component

The system utilizes a data-driven, shift-reduce dependency parser based on maximum entropy models. To simulate varying levels of parser quality, the parser was trained on incremental subsets of domain-specific data, ranging from 100 to 8,000 sentences.

Extraction and Classification

The extraction system employs a Support Vector Machine (SVM) with SubSet Tree Kernels. Rather than relying solely on keywords, the system extracts the "dependency path"—the grammatical route connecting two protein entities within a sentence—and uses this structure as a primary feature for classification.

The dependency path between two protein entities serves as a structural feature for the extraction system.

Datasets

  • GENIA Treebank: 8,000 manually annotated sentences from MEDLINE abstracts, used for parser training.
  • Aimed Corpus: 1,970 sentences from biomedical abstracts used to train and evaluate the PPI extraction system.
  • Penn Treebank (WSJ): Used as an out-of-domain baseline to evaluate the necessity of domain-specific training.

Results and Analysis

The experiments revealed a strong correlation between the quality of the syntactic parser and the performance of the downstream PPI extraction task.

Parser Performance and Downstream Impact

Parser accuracy improved from 72.5% with 100 training sentences to 85.6% with 8,000 sentences. This improvement directly influenced the PPI extraction F-score, which rose from approximately 54 to 57.5.

Comparison of parser accuracy and PPI extraction F-score as a function of training data size.

Key Insights

  1. Domain Specificity: A parser trained on only 500 domain-specific sentences outperformed a parser trained on over 40,000 sentences of general news text (WSJ). This underscores the critical importance of domain-relevant data in specialized fields.
  2. The 1:0.25 Correlation: The authors observed that a 1% absolute improvement in parser accuracy corresponds to roughly a 0.25 point improvement in the PPI extraction F-score.
  3. Diminishing Returns: While parser accuracy rises sharply with the initial 1,000 sentences, it levels off thereafter. Achieving further gains in the final application requires disproportionately large increases in treebank size.

Discussion and Conclusion

The findings suggest that for practical applications, "small and relevant" data is often superior to "large and generic" datasets. Furthermore, the results indicate that when working with a limited annotation budget, resources are likely better spent on task-specific examples (PPI interactions) rather than upstream syntactic trees once a baseline level of parsing proficiency is reached.

While these results are specific to PPI extraction using dependency paths, they highlight the need for a more integrated approach to NLP tool evaluation. Future work should explore more sophisticated interactions between parsers and applications, such as incorporating parser uncertainty or more complex syntactic representations.

Summary updated

BibTeX
@inproceedings{sagae2008evaluating,
  author = {Kenji Sagae and Yusuke Miyao and Rune Saetre and Jun'ichi Tsujii},
  title = {{Evaluating the Effects of Treebank Size in a Practical Application for Parsing}},
  booktitle = {SETQALNLP},
  year = {2008},
  url = {https://openreview.net/forum?id=Cwfi6B7B3Gi},
}
Figure