TL;DR
This paper introduces a novel, modular, and largely unsupervised generative model for coreference resolution. By learning latent entity types and their associated properties, the model significantly improves semantic compatibility, leading to state-of-the-art performance on end-to-end coreference tasks.
What Problem Does This Paper Address?
Coreference resolution, the task of identifying mentions in text that refer to the same real-world entity (e.g., "Barack Obama," "the president," and "he" all referring to the same person), is a fundamental challenge in natural language processing. Existing systems often struggle with semantic compatibility – understanding what kind of entity a mention refers to and whether it aligns with potential antecedents. While syntactic and discourse cues are important, semantic understanding is often the bottleneck, especially when dealing with a vast and nuanced vocabulary. This paper aims to address this by building a model that learns rich semantic representations of entities and their types in a primarily unsupervised manner.
Methods
The authors propose a modular, generative model with three key components:
- Semantic Module: This module learns latent "entity types" (e.g., PERSON, ORGANIZATION, WEAPON) and then generates specific "entities" that inherit from these types. Each entity is characterized by lists of canonical words associated with various properties (like heads, modifiers, governors). This allows for sharing of lexical statistics at the type level, reducing semantic compatibility errors. For instance, a "president" is likely a head for a PERSON type.
- Discourse Module: This module handles the configurational aspects of coreference, determining which entity a mention refers to. It uses a sequential, distance-dependent Chinese Restaurant Process (DD-CRP) to model antecedent choices, incorporating features like sentence distance and syntactic position.
- Mention Module: This module generates the actual words for each mention, drawing from the identified entity's word lists and, if necessary, the broader type distribution. It uses a learned interpolation parameter to balance using entity-specific words versus type-general words.
The entire model is trained using a variational approach, optimizing subgroups of variables in a round-robin fashion. Crucially, the semantic and discourse components are learned in a largely unsupervised manner, with only minimal priming of entity types using prototype lists.
Datasets
The paper uses standard ACE (Automatic Content Extraction) corpora for evaluation:
- Training: A04CU (ACE 2004 newswire), A05ST (ACE 2005 newswire), A05RA (ACE 2005), BLLIP (5k articles), and WIKI (8k abstracts).
- Evaluation: Dev and test sets from A04CU, A05ST, and A05RA.
The authors note that they did not use coreference annotations during training but did use minimal prototype information to prime entity type learning.
Key Insights and Ideas
- Latent Entity Types as a Semantic Bridge: The core idea is that learning abstract "entity types" allows the model to generalize semantic compatibility knowledge across many entities. This is more robust than learning pairwise compatibility between specific words.
- Unsupervised Learning for Semantic Richness: The model's ability to learn rich semantic representations of entities and types without explicit supervision is a significant achievement, enabling it to outperform supervised systems.
- Modularity for Tractability: Breaking down the problem into semantic, discourse, and mention modules makes the complex learning problem more manageable.
- Balancing Entity-Specific and Type-General Knowledge: The mention module's interpolation parameter allows for flexible word generation, capturing both unique entity characteristics and general type properties.
Key Results
The paper reports state-of-the-art results on challenging end-to-end coreference resolution tasks, outperforming previous supervised systems. Specifically:
- On the A05RA dataset, their system achieved a B³ All score of 77.0, surpassing the next best system (Haghighi and Klein, 2009) at 75.4.
- On the A04CU dataset, their system achieved a B³ All score of 67.4, outperforming Stoyanov et al. (2009) at 64.4.
- The model also demonstrates strong performance on the B³ None metric, which is less sensitive to spurious mentions.
- When evaluated as a Named Entity Recognition (NER) system on boundary-annotated mentions, the model achieved 83.2% accuracy, which is notable given the larger inventory of types and the inclusion of nominal mentions.
Limitations and Future Work
- Negative Covariance: The model struggles with mentions that have explicitly contrasting modifiers (e.g., "new president" vs. "old president") because it doesn't explicitly represent negative covariance between such modifiers.
- Spurious Mentions: While the system deterministically extracts mention boundaries, it can extract "spurious" mentions (e.g., events, times) that are not of interest in ACE annotations.
- Pronoun Anaphora: The paper notes that while the model handles pronouns, it might not be as effective as systems specifically designed for pronoun resolution.
- Future Work: The authors suggest exploring more sophisticated ways to model negative covariance and potentially extending the model to handle more complex semantic relationships. They also highlight the potential for the model's type component to be used for unsupervised NER.
Summary updated