Automatic Acquisition of Named Entity Tagged Corpus from World Wide Web

Joohui An, Seungwoo Lee, Gary Geunbae LeeACL 2003

TL;DR

This paper introduces an automated method for creating large, labeled datasets (corpora) for Named Entity Recognition (NER) by leveraging the vastness of the web. By using a list of known entities and a search engine, they can generate a corpus that is comparable in usefulness to manually created ones, significantly reducing the cost and effort of data annotation.

What Problem Does This Paper Address?

Named Entity Recognition (NER) systems, especially those using machine learning, require large amounts of labeled data (corpora) for training. Manually creating these corpora is a time-consuming and expensive process. While the web offers an almost limitless source of text, it's not inherently structured or labeled for NER tasks. This paper addresses the challenge of automatically acquiring a sufficiently large and useful NER-tagged corpus from the web to overcome the data sparseness problem and the high cost of manual annotation.

Methods

The authors propose a four-step process to automatically construct an NE-tagged corpus from the web:

  1. Collecting Web Documents: They use a pre-compiled list of Korean Named Entities (persons, locations, organizations) as queries for a web search engine. A web robot then fetches documents from the search results. Ambiguous entries in the NE list (those that can also be common nouns) are filtered out using a Part-of-Speech dictionary to reduce annotation errors.
  2. Splitting into Sentences: Collected documents are segmented into individual sentences. Sentences that do not contain any target NE instances are discarded. This step focuses on sentence-level annotation because extracting cross-sentence information like co-reference is more complex.
  3. Refining the Web Texts: To improve the quality of the collected data, sentences undergo refinement. This includes:
  4. Separation of functional words (josa): Korean functional words attached to NEs are separated to accurately identify NE boundaries. This is done using a Part-of-Speech tagger.
  5. Segmentation of compound nouns: Compound nouns that might contain an NE as a substring are broken down. If the segmented parts don't match the target NE, the sentence is filtered out.
  6. Verification of usefulness: Sentences are filtered if they lack sufficient context information, even if they contain a correct target NE.
  7. Generating an NE tagged corpus: Finally, the refined sentences are automatically tagged with the appropriate NE category labels. The system annotates one NE instance per sentence.

Datasets

  • Training/Test Data: The paper uses a manually annotated corpus from Seon et al. (2001) for training and testing their NER system.
  • Automatic Corpus: The authors generate their own automatically annotated corpus from the web. This corpus is described as having 68,793 NE instances.
  • NE List: The initial NE list used for web crawling comprises 937 person names, 1,000 locations, and 1,050 organizations.

Key Insights and Ideas

  • Web as an Infinite Resource: The web is presented as a virtually inexhaustible source of linguistic data, making automatic corpus generation a scalable solution.
  • Quality vs. Quantity Trade-off: While automatically generated corpora might have lower accuracy than manually curated ones, their sheer size and ease of expansion can compensate for this, leading to comparable or even superior performance in downstream tasks.
  • Importance of Text Refinement: Simple web crawling and sentence splitting are insufficient. Heuristic-based text refinement (handling functional words, compound nouns, and context) is crucial for improving the quality of the automatically tagged data.
  • Domain Independence: The automatically generated corpus is drawn from the general web, making it more diverse and potentially leading to more robust NER systems compared to systems trained solely on domain-specific manual corpora.

Key Results

  • The automatically acquired NE tagged corpus, when used for training a decision list learning NER system, achieved performance comparable to a manually tagged corpus.
  • Specifically, training with the automatic corpus yielded a precision of 81.45%, recall of 85.41%, and F-measure of 83.43%. This is very close to the performance achieved with a manual corpus (80.21% precision, 86.11% recall, 83.16% F-measure).
  • Combining both manual and automatic corpora for training resulted in a slight improvement (82.03% precision, 85.94% recall, 83.99% F-measure), suggesting the automatic corpus is a valuable supplement.
  • The performance of the NER system improved as the size of the automatically generated corpus increased, with a saturation point observed around 1.2 million words for the 'person' category.

Limitations and Future Work

  • Limitations:
    • The accuracy of the automatically tagged corpus is acknowledged to be generally lower than manually tagged corpora.
  • The paper focuses on only three major NE categories (person, organization, location) due to the perceived shortage of tagged data for these.
  • The automatic corpus is noted to have less lexical and contextual information compared to a manual corpus of the same size.
  • Future Work:
    • The authors plan to apply more sophisticated natural language processing schemes to further improve the accuracy of automatic NE tagged corpus generation.

Summary updated

BibTeX
@inproceedings{an2003automatic,
  author = {Joohui An and Seungwoo Lee and Gary Geunbae Lee},
  title = {{Automatic Acquisition of Named Entity Tagged Corpus from World Wide Web}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2003},
}
Figure