A Methodology for Extrinsically Evaluating Information Extraction Performance

Michael Crystal, Alex Baron, Katherine Godfrey, Linnea Micciulla, Yvette Tenney, Ralph WeischedelEMNLP 2005

TL;DR

This paper introduces a novel methodology for evaluating Information Extraction (IE) systems by measuring their impact on a downstream task, specifically Question Answering (QA). They developed a "blending" technique to create intermediate IE performance levels, allowing them to study how improvements in IE accuracy translate to gains in QA speed and accuracy, thereby providing a more practical measure of IE utility.

What Problem Does This Paper Address?

Traditional evaluations of Information Extraction (IE) systems often focus on "intrinsic" measures, like precision and recall of extracted entities and relations. While useful, these metrics don't directly tell us how much better an IE system makes a real-world application. For instance, a slight improvement in IE precision might have a huge impact on a user's ability to quickly find answers to questions, or it might have negligible effect. The authors highlight the lack of robust "extrinsic" evaluations that measure the utility of IE in a task. They aim to bridge this gap by developing a method to quantify the impact of IE performance on a downstream task (QA) and to explore hypothetical scenarios, such as "what if IE accuracy were X% better?"

Methods

The core of their methodology is a novel database blending strategy. They start with two extreme IE databases: one from their state-of-the-art automated IE system (SERIF, representing 0% human accuracy) and one from manual human annotation (representing 100% human accuracy). They then create intermediate databases by merging portions of these two. For example, a "33% blend" database would contain 33% of the entity mentions from the human database and 67% from the machine database, along with corresponding relationships and co-reference links.

This blending technique allows them to create a continuum of IE quality. They then use these blended databases in an extrinsic evaluation: a Question Answering (QA) task. Participants answer questions using a QA interface (FactBrowser) that accesses the information from these varying quality IE databases. The key measures of effectiveness are QA accuracy and time to complete the QA task. By having participants use different blended databases, they can observe how changes in IE performance affect QA performance.

Datasets

The study utilizes a dataset of 946 broadcast news and newswire articles that were previously annotated for entities, relations, and within-document co-reference as part of recent Automatic Content Extraction (ACE) evaluations. This existing annotation served as the gold standard (human-quality database). The automated IE system, SERIF, was run on these same documents to generate the machine-quality database. No new benchmark datasets were introduced; rather, the existing annotated data was leveraged to create the blended IE databases.

Key Insights and Ideas

  • Bridging the Intrinsic-Extrinsic Gap: The blending methodology is a clever way to simulate varying levels of IE performance without needing to retrain or significantly alter the IE system itself. This makes it a practical tool for understanding the value of IE improvements.
  • "A Little Human Goes a Long Way": The results suggest that even a small increase in IE quality (moving from machine-generated to a low percentage of human-like data) can lead to significant improvements in QA performance. This implies that focusing on improving the most critical or error-prone aspects of IE might yield substantial downstream benefits.
  • QA as a Proxy for IE Utility: The study demonstrates a strong correlation between IE performance (as represented by the blend percentage) and QA performance (accuracy and speed). This validates QA as a suitable downstream task for evaluating the practical utility of IE systems.
  • Hypothetical Performance Questions: The methodology directly addresses the paper's goal of answering "what if" questions. By interpolating IE quality, they can estimate the performance gains achievable with hypothetical improvements to IE systems.

Key Results

  • Improved QA Accuracy and Speed: As the IE database quality increased (moving from machine to human annotation), QA accuracy consistently improved, and the time taken to answer questions decreased.
  • Significant Impact of IE Quality: In broader experiments (e.g., 0-33-67-100% blends), the impact of IE database quality on QA performance and time was highly significant (P < 0.0001).
  • Threshold for Improvement: For a target QA accuracy of 70%, the study estimated that a database blend between 41% and 46% machine extraction was required.
  • Diminishing Returns in Narrower Ranges: In Experiment 3, which examined smaller increments in IE quality (33-41-50-58%), the impact on QA performance was not statistically significant. This suggests that beyond a certain point, small improvements in IE quality might not translate to noticeable gains in the downstream task, or that more participants would be needed to detect such subtle differences.

Limitations and Future Work

  • Limited Scope of Downstream Task: The study focused solely on Question Answering. The authors acknowledge that the impact of IE quality might differ for other downstream tasks.
  • Atypical Participants: Experiment 3 encountered participants with atypical QA patterns, which may have masked statistically significant differences. Larger studies could help clarify whether these were outliers or representative of a subpopulation.
  • Generalizability of Blending: While the blending method is effective, its specific implementation and the resulting "value scores" might be dataset and task-dependent.
  • Future Work: The authors suggest that this methodology can be employed for further controlled experiments to explore the relationship between IE and other tasks, and to better understand the specific types of IE improvements that yield the greatest downstream benefits. They also note the potential for larger experiments to reveal more subtle differences.

Summary updated

BibTeX
@inproceedings{crystal2005methodology,
  author = {Michael Crystal and Alex Baron and Katherine Godfrey and Linnea Micciulla and Yvette Tenney and Ralph Weischedel},
  title = {{A Methodology for Extrinsically Evaluating Information Extraction Performance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
}
Figure