Name Perplexity

Octavian PopescuNAACL 2009

TL;DR

This paper introduces "name perplexity," a statistical measure to quantify how ambiguous a person's name is in a given text corpus. By categorizing names based on their perplexity, the authors propose a method to dynamically adjust the amount of contextual information needed for coreference resolution, leading to a better balance between precision and recall in identifying mentions of the same person across documents.

What Problem Does This Paper Address?

Identifying all mentions of a specific person across a large collection of documents (Person Cross-Document Coreference or PCDC) is a notoriously difficult task in Natural Language Processing. A key challenge is determining how much contextual information is "enough" to confidently link two name mentions. This amount of context varies greatly depending on the name itself and the diversity of the corpus. For instance, a common name like "John Smith" might appear for many different individuals, requiring extensive context to distinguish them. Conversely, a rare name like "Barack Obama" is likely to refer to the same person across documents, potentially needing less context. Existing systems often struggle to adapt this context requirement dynamically, leading to either too much context (and potential errors) or too little (and missed coreferences).

Methods

The core contribution of this paper is the concept of name perplexity. This is a statistical measure that quantifies the ambiguity of a name within a corpus.

  • One-Token Name Perplexity: For single-word names (e.g., "John"), perplexity is defined as the number of different other words or entities it's associated with in the corpus. For example, if "John" appears with "Smith," "F. Kennedy," and "Travolta," its perplexity is 3.
  • Two-Token Name Perplexity: For multi-word names (e.g., "John Smith"), the perplexity is derived from the perplexity of its constituent one-token names. The paper proposes heuristics for combining these, including averaging the perplexity classes or considering it one class lower if the individual token perplexities are the same.
  • Perplexity Classes: Names are then grouped into categories (very low, low, medium, high, very high) based on their perplexity.
  • Prior Coreference Probability: The paper hypothesizes that lower perplexity correlates with a higher prior probability of coreference. This means names with low perplexity are more likely to refer to the same person.
  • Dynamic Context Adjustment: The key idea is to use these perplexity classes to inform a PCDC system. If a name has a low perplexity (high prior coreference probability), the system can require less contextual evidence to establish a coreference. Conversely, for high-perplexity names, more contextual evidence will be needed.
  • Statistical Model for Class Delimitation: The authors propose a statistical method to determine the boundaries between perplexity classes. This method aims to ensure that within each class, the prior coreference probability has low variance, maximizing confidence in the estimates.

Datasets

The experiments and examples in this paper utilize the Adige500k corpus, an Italian news corpus. The authors mention that this corpus contains 106,192 different one-token names, 429,251 different two-token names, and 36,773 three-token names.

Key Insights and Ideas

  • Name ambiguity is a fundamental property of the corpus, not just the name itself: The "John Smith" example highlights that the diversity of the corpus (how many different "John Smiths" exist) is more critical than the specific relationship "works for U.N." in determining context relevance.
  • Prior probability of coreference is a powerful signal: Instead of solely relying on analyzing contextual evidence, knowing the inherent likelihood of two names referring to the same person can significantly simplify the coreference task.
  • Perplexity offers a quantifiable measure of name ambiguity: This provides a systematic way to categorize names and tailor coreference strategies.
  • Balancing precision and recall is crucial: The paper emphasizes that name perplexity helps achieve a better trade-off between correctly identifying coreferences (precision) and finding all possible coreferences (recall).

Key Results

The paper presents preliminary experimental results using the Adige500k corpus.

  • Distribution of Names into Perplexity Classes:
    • First Names: Very low (37.5%), Low (27.6%), Medium (20.9%), High (8.7%), Very high (5.3%).
  • Last Names: Very low (57.02%), Low (20.31%), Medium (17.51%), High (3.36%), Very high (1.8%).
  • General Observation: Approximately 60% of two-token names fall into "low" or "very low" perplexity classes.
  • Impact on Coreference:
    • For "low" and "very low" perplexity classes, the system can adequately reduce the amount of contextual evidence required.
  • For "very high" perplexity classes, requiring stronger similarity between contexts led to increased precision.
  • The authors note that coreferring on the basis of just one special word and one named entity for "low" or "very low" perplexity names resulted in a loss of only ~1.5% precision but gained up to 40% in recall.

Limitations and Future Work

  • Problematic Cases: The paper acknowledges that cases where the perplexity class is "very high" and the number of occurrences is also very large remain challenging.
  • Future Work:
    • Increase the sample size used to delimit perplexity classes to improve confidence.
  • Further develop the statistical equation (2) to precisely determine the number of required cases for each perplexity class.
  • The authors' first objective is to solve the most problematic cases identified by their approach.

Summary updated

BibTeX
@inproceedings{popescu2009name,
  author = {Octavian Popescu},
  title = {{Name Perplexity}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure