Crowdsourcing the evaluation of a domain-adapted named entity recognition system

Asad B. Sayeed, Timothy J. Meyer, Hieu C. Nguyen, Olivia Buzek, Amy WeinbergNAACL 2010

TL;DR

This paper tackles the challenge of adapting Named Entity Recognition (NER) systems to new domains, which is crucial for their real-world utility. They demonstrate that a simple, post-hoc rule-based error reduction strategy, evaluated using crowdsourcing, can significantly improve NER precision with minimal effort and cost, even on unseen data.

What Problem Does This Paper Address?

High-performing Named Entity Recognition (NER) systems, often built using machine learning, struggle when applied to data from domains that differ from their training data. Adapting these systems typically requires generating new, fully annotated training data, which is a costly and time-consuming process. This paper addresses the need for a more efficient and cost-effective way to evaluate and improve NER system performance in new domains, specifically focusing on reducing errors without the burden of extensive retraining.

Methods

The authors employ a two-pronged approach:

  1. Domain-Specific Error Reduction: They first analyze the errors made by a pre-trained NER system (Identifinder 3.3) on a new domain (IT business press). Based on this analysis, they develop a set of simple, rule-based filters to correct common error types. These rules leverage techniques like part-of-speech tagging and majority voting across the corpus to identify and fix misclassifications and extraneous entity inclusions.
  2. Crowdsourced Evaluation: To validate the effectiveness of their error reduction strategy, they utilize crowdsourcing, specifically Amazon Mechanical Turk (MTurk). They design a task where crowdworkers (Turkers) are presented with NER system output and asked to label entities as "Person," "Organization," "Neither," or "Don't Know." This crowdsourced data serves as a ground truth for evaluating both the original NER system and the system after applying their rule-based filters.

Datasets

  • InformationWeek: A collection of 200 articles from the IT business press was used for initial expert annotation and evaluation.
  • InformationWeek (extended): An additional 800 articles from the same publication were used for further evaluation, primarily with crowdsourced annotations.

The paper effectively creates a new benchmark for evaluating domain adaptation of NER systems by using crowdsourced annotations as a validation set.

Key Insights and Ideas

  • Domain Adaptation is a Major Hurdle: Even state-of-the-art NER systems exhibit significant performance drops when applied to out-of-domain data.
  • Post-Hoc Rules Offer Significant Gains: Simple, expert-designed rules can achieve substantial precision improvements (around 10%) without requiring full retraining.
  • Crowdsourcing is a Viable Evaluation Tool: MTurk can be effectively used to generate reliable evaluation data for NER, even for tasks requiring some level of judgment, by carefully designing simple tasks and using multiple annotators per item.
  • Precision Over Recall is Often Key: For applications like opinion mining, reducing false positives (incorrectly identified entities) is more critical than missing some entities, making precision improvements highly valuable.

Key Results

  • The rule-based error reduction system increased the precision of the Identifinder system from 0.74 to 0.86 on the expert-annotated IT domain data, with only a minimal sacrifice in recall.
  • This improvement was maintained on the crowdsourced data, with precision rising from 0.67 to 0.77 on the 800-document set.
  • The entire process, including expert annotation, rule design, and crowdsourcing setup, took an estimated two person-weeks.
  • Crowdsourced annotations showed substantial agreement with expert annotations, with a Cohen's κ of 0.68.

Limitations and Future Work

  • Single Domain and Publication: The study was conducted on a single publication within the IT domain. Future work should explore reliability across different publications and domains.
  • Optimal Annotator Number: The paper used three annotators per HIT. Further research is needed to determine the optimal number of annotators for robust cross-domain results.
  • Comparison with Retraining: While the paper highlights the cost-effectiveness of their approach, a direct comparison with the cost and performance of retraining an NER system for a new domain would be valuable.
  • Advanced Techniques: The crowdsourced data generated can serve as a source for training more advanced domain adaptation techniques if needed.

Summary updated

BibTeX
@inproceedings{sayeed2010crowdsourcing,
  author = {Asad B. Sayeed and Timothy J. Meyer and Hieu C. Nguyen and Olivia Buzek and Amy Weinberg},
  title = {{Crowdsourcing the evaluation of a domain-adapted named entity recognition system}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure