Fine Grained Classification of Named Entities

Michael Fleischman, Eduard HovyCOLING 2002

TL;DR

This paper tackles the challenge of classifying person names into finer-grained subcategories beyond the standard "person" tag. They introduce a supervised learning method that leverages local context, global semantic information from topic signatures and WordNet, and a novel algorithm to handle name variations, achieving promising results.

What Problem Does This Paper Address?

Current Named Entity Recognition (NER) systems are adept at broad categorization (person, organization, location), but this often lacks the granularity needed for advanced applications like Question Answering or Ontology generation. This paper focuses on the "person" category, aiming to classify individuals into more specific roles (e.g., politician, athlete, entertainer). This is a difficult task because person names can be ambiguous (the same name referring to different people) and often appear in various forms (e.g., "Ronald Reagan" vs. "Mr. Reagan"). Unlike location names, exhaustive lists of person names by category are not readily available, making training data generation a significant hurdle.

Methods

The authors propose a supervised learning approach that combines several feature types:

  • Word Frequency Features: These capture the frequency of words surrounding a person's name, indicating their co-occurrence with specific subcategories in the training data. They consider unigrams, bigrams, and trigrams in the immediate vicinity of the name.
  • Topic Signature Features: These leverage semantic information by using topic signatures derived from the training data. A topic signature is a list of terms that signal membership in a relevant topic or category, with scores indicating their importance. The authors use a likelihood ratio (λ-score) to compute these signatures and weight them by the inverse square of their distance from the instance.
  • WordNet Features: To address the limitation of topic signatures not capturing synonyms or related terms, they incorporate WordNet. They expand the WordNet hypernym tree for words surrounding the instance and assign scores based on the topic signature database, weighted by the inverse of their height in the tree.

A key innovation is the MemRun algorithm, designed to handle orthographic inconsistencies and improve classification by considering multiple occurrences of the same person's name within a text. It operates in two rounds: first, it generates a hypothesis for each instance; second, it standardizes subcategorizations based on the most frequent classification with high confidence, effectively averaging out individual classification uncertainties.

Datasets

  • Training Data: A large corpus of person instances was compiled from a TREC9 database, specifically articles from the Associated Press (AP) and the Wall Street Journal (WSJ). This data was tokenized, stemmed, part-of-speech tagged, and named entity tagged. A seed dataset of 100 instances per category was hand-tagged, and a bootstrapping procedure was used to generate a large training set (approximately 25,000 instances) with high confidence.
  • Validation Set: A subset of 1000 instances from the generated training set was hand-tagged for validation.
  • Held-out Test Set: A separate set of 1300 instances, also from the AP and WSJ, was collected and hand-tagged to evaluate the generalizability of classifiers trained on the automatically generated data.

Key Insights and Ideas

  • The Importance of Context: The paper strongly emphasizes that classifying person names requires more than just local word frequencies; global semantic information from topic signatures and WordNet is crucial.
  • Handling Name Variations is Key: The MemRun algorithm's success in improving performance highlights the critical need to account for different ways a person can be referred to within a document.
  • Data Generation is a Major Hurdle: The authors acknowledge that their automatic data generation method, while effective in producing a high-quality training set, results in a skewed distribution (more easy-to-classify instances, fewer partial names) compared to real-world data, impacting generalizability.
  • WordNet's Potential and Pitfalls: While WordNet features improve coverage, they can also introduce noise due to word senses and generality, suggesting a need for careful tuning or filtering.
  • C4.5's Robustness: The C4.5 decision tree algorithm consistently performed well, especially when combined with the MemRun algorithm, demonstrating its effectiveness for this task.

Key Results

  • The proposed method, using a combination of word frequency, topic signature, and WordNet features, significantly outperforms a baseline of always choosing the most frequent category.
  • The MemRun algorithm demonstrably improves classification performance on both validation and held-out test sets, indicating its effectiveness in handling name variations and improving robustness.
  • In Experiment 2, the C4.5 decision tree algorithm achieved the highest accuracy (70.4% on the validation set) among the tested learning algorithms, outperforming k-Nearest Neighbors, Naïve Bayes, Support Vector Machines, and Neural Networks.
  • Experiment 3 showed that individual feature sets (frequency, topic signature, WordNet) were sufficient to outperform the baseline, with topic signature features performing slightly better than WordNet features. Combining all three feature sets yielded the best results.
  • A significant performance gap was observed between the validation set and the held-out test set, underscoring the impact of the biased training data.

Limitations and Future Work

  • Training Data Bias: The authors explicitly state that their training data is skewed, containing a disproportionate number of easy-to-classify instances and limited coverage of certain categories (e.g., lower-level businesspersons). This leads to a performance disparity on the held-out test set.
  • Handling Miscellaneous Classifications: The system needs to address instances that don't fit neatly into any category or might belong to multiple categories (e.g., Ronald Reagan as both politician and entertainer).
  • Extending to Other Classes: The current work focuses on persons; future work will involve extending the system to other entity types like organizations.
  • Future Work Suggestions:
    • Investigate more sophisticated bootstrapping methods (e.g., co-training, co-testing).
  • Adapt hierarchical decision list algorithms for word sense disambiguation.
  • Explore ensemble learning to capitalize on different feature sets.
  • Refine WordNet feature usage, potentially by limiting the depth of hypernyms considered.
  • Address the challenges of miscellaneous classifications and multi-category instances.

Summary updated

BibTeX
@inproceedings{fleischman2002fine,
  author = {Michael Fleischman and Eduard Hovy},
  title = {{Fine Grained Classification of Named Entities}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2002},
}
Figure