An Approach to Text Corpus Construction which Cuts Annotation Costs and Maintains Reusability of Annotated Data

Katrin Tomanek, Joachim Wermter, Udo HahnEMNLP 2007

TL;DR

This paper introduces an Active Learning (AL) approach for constructing annotated text corpora that significantly reduces annotation costs (up to 72%) without sacrificing data quality. Crucially, it demonstrates that corpora annotated using this AL method are reusable for training classifiers different from the ones used during the annotation process, a key challenge in AL research.

What Problem Does This Paper Address?

Creating high-quality annotated text corpora is a fundamental but costly bottleneck in Natural Language Processing (NLP). While techniques like syntax tagging have matured, semantic annotations (like named entities, events, etc.) are highly domain-dependent, requiring new annotations for each new domain. This leads to a continuous need for efficient annotation methods. Active Learning (AL) offers a promising solution by allowing the learning model to select the most informative data points for human annotation, thereby minimizing the amount of data that needs to be labeled. However, a significant challenge with AL is that the data selected is often highly tuned to a specific classifier and feature set, raising concerns about the reusability of the annotated corpus for training different models or using alternative feature sets. This paper aims to address both the cost reduction and reusability aspects of AL for corpus construction.

Methods

The authors propose using AL not just as a learning protocol but as a corpus annotation method. Their approach is based on Query-by-Committee (QBC), a heuristic AL strategy known for its computational tractability. In QBC, a "committee" of classifiers is trained on the already annotated data. This committee then predicts labels for a pool of unannotated data. Examples where the committee members disagree the most are selected for human annotation. This disagreement is measured using vote entropy, which quantifies the uncertainty among the classifiers.

Key methodological contributions include:

  • QBC for Corpus Annotation: Adapting QBC, typically used for model training, to guide the annotation process itself.
  • Committee of Maximum Entropy (ME) Classifiers: Using ME classifiers for the AL committee due to their computational efficiency, which is crucial for fast selection cycles in an interactive annotation process.
  • Adaptive Stopping Criterion: Proposing an adaptive stopping criterion based on the disagreement among committee members approaching zero. This indicates that further annotations are unlikely to improve the model significantly.
  • Reusability Experiments: Designing experiments to test if corpora annotated using their AL method (with ME classifiers and specific features) can be effectively used to train different classifiers (like Conditional Random Fields - CRFs) and with different feature sets.

Datasets

The authors evaluated their approach on two common entity-annotated corpora:

  • CoNLL-2003: A general-language newspaper domain corpus with three entity types (persons, locations, organizations).
  • PENNBIOIE corpus: A sublanguage corpus from the biology domain, specifically the oncology part. This was split into two subcorpora for experiments:
  • PBgene: Focusing on gene entity subtypes.
  • PBvar: Focusing on variation entity subtypes.

For evaluation, they used the CoNLL training set for AL and its test set as gold standard. For PBgene and PBvar, they randomly split the corpora into 90% for AL simulation and 10% as gold standard.

Key Insights and Ideas

  • AL as an Annotation Engine: The core idea is to leverage AL's ability to identify informative examples to build a high-quality annotated corpus more efficiently than random selection.
  • QBC's Practicality: QBC is favored over other AL methods due to its computational efficiency, making it suitable for interactive annotation where annotator idle time must be minimized.
  • Disagreement as a Signal: The level of disagreement among committee members is a robust indicator of annotation progress and can serve as an effective stopping criterion, avoiding the need for a separate, manually annotated gold standard for evaluation during the process.
  • Reusability is Achievable: Contrary to some prior work, this paper provides strong evidence that AL-annotated corpora are indeed reusable. Even when the AL committee uses a simpler classifier (ME) and a specific feature set, the resulting corpus can be effectively used to train more complex classifiers (CRF) with potentially richer feature sets.
  • Sparsity Benefits AL: AL is particularly beneficial for annotating sparse entity types, as it can achieve a much higher density of relevant examples in the annotated corpus compared to random selection.

Key Results

  • Annotation Cost Reduction: The AL approach achieved significant reductions in annotation effort, ranging from 48% to 72% across the tested corpora, compared to random selection.
  • Quality Maintained: These cost reductions were achieved without sacrificing annotation quality, as measured by f-scores.
  • Adaptive Stopping Criterion Effectiveness: The disagreement-based stopping criterion was shown to be a valid and practical method for determining when to terminate AL.
  • Algorithm Flexibility: Corpora annotated using ME classifiers for AL could be effectively used to train CRF testers, with minimal performance degradation.
  • Feature Flexibility: Corpora annotated with a subset of features were still reusable by testers with more comprehensive feature sets, indicating robustness.
  • Sparsity Impact: For sparse entity types (like cytokine receptors), AL created an annotated corpus with significantly higher entity density (up to 15x) compared to gold standard annotations derived from random selection.

Limitations and Future Work

  • Direct Comparison of AL Strategies: The authors acknowledge that a direct comparison between QBC and uncertainty sampling for data reusability is needed.
  • Problem Class Influence: Further research is required to investigate whether the problem class (classification vs. ranking) influences data reusability.
  • Real-World Annotation Project: While the paper reports on simulations and ongoing real-world annotation, a more in-depth analysis of the practical challenges and benefits in a large-scale, continuous annotation project could be beneficial.
  • Feature Set Optimization: While the paper shows reusability with modified feature sets, exploring optimal strategies for feature selection during AL and subsequent testing remains an area for future work.

Summary updated

BibTeX
@inproceedings{tomanek2007approach,
  author = {Katrin Tomanek and Joachim Wermter and Udo Hahn},
  title = {{An Approach to Text Corpus Construction which Cuts Annotation Costs and Maintains Reusability of Annotated Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure