Specialized Models and Ranking for Coreference Resolution

Pascal Denis, Jason BaldridgeEMNLP 2008 · OpenReview

Abstract

This paper investigates two strategies for improving coreference resolution: (1) training separate models that specialize in particular types of mentions (e.g., pronouns versus proper nouns) and (2) using a ranking loss function rather than a classification function. In addition to being conceptually simple, these modifications of the standard single-model, classification-based approach also deliver significant performance improvements. Specifically, we show that on the ACE corpus both strategies produce f-score gains of more than 3% across the three coreference evaluation metrics (MUC, B3, and CEAF).

Introduction

Coreference resolution is the fundamental NLP task of determining which "mentions" in a text (such as "Barack Obama," "the president," and "he") refer to the same real-world entity. Historically, this task has been framed as a binary classification problem: given a pair of mentions, the system decides whether they corefer. However, this traditional approach suffers from two significant limitations. First, it treats every pair in isolation, ignoring the competitive nature of antecedent selection. Second, it typically employs a monolithic model that applies the same rules to all mention types, despite the vastly different linguistic behaviors of pronouns versus proper names.

This research from the University of Texas at Austin and INRIA explores how two conceptually simple changes—specializing models by mention type and switching from classification to ranking—can significantly boost performance.

Methodology

From Classifiers to Rankers

The authors propose moving away from binary "Yes/No" classifiers toward a log-linear ranking model. When the system encounters a mention, the ranker evaluates all preceding candidate mentions simultaneously and assigns a probability to each. This approach better reflects the linguistic reality where a mention "competes" for the best antecedent among several candidates.

Specialized Models

Rather than a single monolithic model, the problem is decomposed into five specialized sub-models based on the linguistic form of the mention:

  • Third-person pronouns (3P): e.g., he, she, it
  • Speech pronouns (SP): e.g., I, you
  • Proper names (PN): e.g., Pascal Denis
  • Definite NPs (Def-NP): e.g., the paper
  • Others (Oth): Indefinite or bare noun phrases

Each model utilizes a tailored set of features. For instance, the "Proper Name" model relies heavily on string similarity and acronym detection, whereas the "Pronoun" model focuses on sentence distance and morphosyntactic agreement.

Feature selection across specialized mention types.

The Anaphoricity Filter

Because a ranker is designed to always select a "winner," it requires a "gatekeeper" to handle mentions that do not refer back to anything. The authors implement a discourse-status classifier to determine if a mention is "discourse-new" (introducing a new entity) or "discourse-old" (referring back to a previous mention). Only "discourse-old" mentions are passed to the ranking engine.

Experimental Results

The system was evaluated on the ACE (Automatic Content Extraction) Phase 2 corpus using three standard metrics: MUC, $B^3$, and CEAF.

The results demonstrate that the combined system (RANK+DS+SP), which utilizes both discourse status (DS) and specialized models (SP) within a ranking framework, significantly outperforms the baseline monolithic classifier across all metrics.

Performance comparison of classification and ranking models across standard metrics.

Key insights from the evaluation include:

  • Ranking captures competition: By evaluating the entire candidate set at once, the model distinguishes the "best" match from merely "good" matches.
  • Linguistic specialization reduces noise: Features like "string match" can actually degrade pronoun resolution performance if included in a monolithic model, but are vital for proper names. Specialization allows each model to focus on relevant signals.
  • Efficiency: While recent trends favored complex "global" models, this research shows that refining "local" models with ranking and specialization achieves competitive results with lower computational complexity ($O(n^2)$ vs $O(n^3)$).

Conclusion and Future Work

The proposed modifications yielded F-score improvements of over 3% on the ACE corpus. However, the system's performance remains constrained by the "discourse-status" filter; if a mention is incorrectly labeled as "new," the ranker never processes it. "Oracle" tests assuming a perfect filter suggest potential F-scores in the high 70s or 80s, indicating that improving initial anaphoricity detection is a primary avenue for future research. Additionally, future work could explore varying search windows for different mention types to further optimize the ranking process.

Summary updated

Abstract

This paper investigates two strategies for improving coreference resolution: (1) training separate models that specialize in particular types of mentions (e.g., pronouns versus proper nouns) and (2) using a ranking loss function rather than a classification function. In addition to being conceptually simple, these modifications of the standard single-model, classification-based approach also deliver significant performance improvements. Specifically, we show that on the ACE corpus both strategies produce f-score gains of more than 3% across the three coreference evaluation metrics (MUC, B3, and CEAF).
BibTeX
@inproceedings{denis2008specialized,
  author = {Pascal Denis and Jason Baldridge},
  title = {{Specialized Models and Ranking for Coreference Resolution}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
  url = {https://openreview.net/forum?id=r1ZdIzzOZr},
}
Figure