Exploiting Wikipedia as External Knowledge for Named Entity Recognition

Jun’ichi Kazama, Kentaro TorisawaEMNLP 2007

TL;DR

This paper proposes a simple yet effective method to improve Named Entity Recognition (NER) by leveraging Wikipedia. They extract category labels from the first sentence of Wikipedia articles and use these as features in a CRF-based NER model, achieving significant accuracy improvements over baseline methods.

What Problem Does This Paper Address?

Named Entity Recognition (NER) systems aim to identify and classify named entities (like people, organizations, locations) in text. A common approach to boost NER performance is by using gazetteers, which are lists of known entities. However, building and maintaining these gazetteers is a labor-intensive and time-consuming task. While Wikipedia is a vast and structured source of information about named entities, it's not directly usable as a gazetteer. This paper addresses the challenge of efficiently extracting useful knowledge from Wikipedia to enhance NER systems without the burden of manual gazetteer creation.

Methods

The core of the proposed method involves extracting "category labels" from Wikipedia articles. For a given candidate word sequence (potential named entity), the system first finds the corresponding Wikipedia article. If the article is a redirect, it follows the redirection until a primary article is found. The category label is then extracted from the first sentence of this article, which typically serves as a definition. Specifically, the method focuses on the head noun of the noun phrase following "is" or "was" in the definition sentence. These extracted category labels are then used as features in a Conditional Random Field (CRF) based NER tagger. The CRF model learns to associate these Wikipedia category labels with the named entity tags (e.g., PER, LOC, ORG).

Datasets

The experiments were conducted using the CoNLL 2003 NER dataset. This dataset consists of English newspaper articles annotated with four entity categories: PER (Person), LOC (Location), ORG (Organization), and MISC (Miscellaneous). The dataset was split into training, development, and testing sets.

Key Insights and Ideas

  • Wikipedia's first sentence is a rich source of entity definition: The initial sentence of a Wikipedia article, often acting as a definition, contains valuable information about the entity's category.
  • Simple extraction yields significant gains: A straightforward method of extracting the head noun from the definition sentence as a category label is surprisingly effective for improving NER.
  • Category labels are more impactful than simple matching: While matching entity names directly from Wikipedia (similar to a gazetteer) provides some improvement, using the category labels derived from Wikipedia offers a more substantial boost in NER accuracy.
  • Wikipedia complements existing gazetteers: Combining Wikipedia features with traditional gazetteer features leads to further improvements, suggesting that Wikipedia contains entities not present in standard gazetteers.
  • The method is robust across entity types and metrics: The improvements observed were consistent across different entity categories (PER, LOC, ORG, MISC) and evaluation metrics (precision, recall, F-measure).

Key Results

The paper demonstrates that incorporating Wikipedia category labels as features significantly improves NER performance.

  • The baseline model achieved an F-measure of 84.99% on the CoNLL 2003 dataset.
  • Adding Wikipedia category label features (model (D):
  • wp_m, wp_c) improved the F-measure to 86.57%, a gain of 1.58 points over the baseline.
  • Combining Wikipedia features with gazetteer features (model (E)) further boosted performance to 87.67% F-measure.
  • The final model, incorporating both gazetteer and Wikipedia features, including word combinations (model (F)), achieved an F-measure of 88.02%, a substantial 3.03 point improvement over the baseline.
  • The study also showed that Wikipedia features provided a noticeable improvement even with smaller training datasets, and their effect became more pronounced as the training data size decreased.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Disambiguation of ambiguous entities: The current method does not explicitly handle disambiguation pages or entities with multiple meanings. While the current approach benefits from Wikipedia's tendency to prioritize the most common meaning, more sophisticated disambiguation techniques are needed for further improvements.
  • Exploiting other Wikipedia structures: The paper primarily focuses on the first sentence. Future work could explore other Wikipedia structures like categories, infoboxes, and internal links for richer knowledge extraction.
  • Fine-grained NER: The current approach is evaluated on abstract entity categories. Investigating its effectiveness for more fine-grained NER categories (e.g., distinguishing between different types of musicians or athletes) is a promising direction.
  • Handling incorrectly extracted categories: While combining features helped, the paper notes that incorrectly extracted categories can still be an issue. Investigating methods to better handle or filter these incorrect extractions is important.

Summary updated

BibTeX
@inproceedings{kazama2007exploiting,
  author = {Jun’ichi Kazama and Kentaro Torisawa},
  title = {{Exploiting Wikipedia as External Knowledge for Named Entity Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure