Stanford's Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task.

Heeyoung Lee, Yves Peirsman, Angel X. Chang, Nathanael Chambers, Mihai Surdeanu, Dan JurafskyCoNLL Shared Task 2011 · OpenReview

Abstract

Stanford’s winning entry for the CoNLL-2011 coreference resolution task demonstrates that a "sieve" of deterministic, rule-based models—applied in order from highest to lowest precision—can outperform complex machine learning approaches. By propagating information like gender and number across entity clusters, the system makes increasingly informed decisions as it moves from simple string matching to difficult pronoun resolution.

Introduction

Coreference resolution is the task of identifying all expressions (mentions) in a text that refer to the same real-world entity. This task is notoriously difficult because it requires a mix of syntactic rules, semantic knowledge (e.g., knowing that a "plane" is an "aircraft"), and discourse understanding. Many systems struggle to balance precision (avoiding incorrect links) and recall (finding all correct links). The authors aim to build a robust pipeline that handles these diverse challenges without the need for complex, often opaque, global optimization models.

Methodology

The Multi-Pass Sieve Architecture

The core of the system is the Multi-Pass Sieve. Instead of making all coreference decisions at once, the system applies 13 distinct "sieves" in a specific order, starting with very "safe" rules such as Exact String Match. Once two mentions are linked into a cluster, their attributes (like gender, number, and animacy) are shared across the entire cluster. This Attribute Propagation is a key innovation: if one mention in a cluster is identified as "singular" and "male," every other mention in that cluster inherits those traits, helping subsequent, "riskier" sieves (like pronoun resolution) avoid mistakes.

Discourse and Semantic Extensions

The system incorporates specialized sieves to handle complex linguistic phenomena:

  • Discourse Sieve: Uses speaker information to resolve pronouns. By identifying the speaker of a quote, the system can confidently link "I" to the speaker and "you" to the listener.
  • Semantic Sieves: Utilizes external knowledge bases like WordNet, Wikipedia, and Freebase to identify aliases (e.g., "America Online" and "AOL") or lexical chains (e.g., "Britain" and "country").

Experimental Evaluation

The system was evaluated on the CoNLL-2011 Shared Task dataset (OntoNotes corpus), covering news, magazine articles, and broadcast conversations. Evaluation was split into a Closed Track (limited to provided data) and an Open Track (allowing external knowledge sources).

Performance Analysis

An ablation study highlights the impact of different components. The discourse sieve provided a significant boost (nearly 2 points), while semantic sieves showed mixed results, sometimes introducing noise by linking incorrectly detected mentions.

Ablation study of system components showing the impact of discourse (D), semantic (S), and gold mentions (GM).

Results

Stanford’s system ranked first in both the open and closed tracks of the CoNLL-2011 Shared Task. The official results demonstrate the system's robustness across different mention boundary conditions.

Official CoNLL-2011 Shared Task results for Closed and Open tracks with varying mention boundary conditions.

Discussion

The Mention Detection Bottleneck

A critical finding was the massive performance gap—approximately 15 points—between using "gold" (human-annotated) mentions versus "predicted" (machine-generated) mentions. This suggests that a major hurdle in coreference resolution is not just the linking logic, but the initial identification of valid mentions.

Simplicity and Precision

Despite the trend toward complex joint-inference models, this work proves that a well-engineered pipeline of simple deterministic rules can achieve state-of-the-art results. By prioritizing precision-first architecture, the system builds a reliable foundation of entity clusters that reduces the search space for harder cases later in the pipeline.

Conclusion

The Multi-Pass Sieve system demonstrates that modularity and attribute propagation are powerful tools for coreference resolution. Future work may focus on "joint modeling" of mentions and coreference chains to overcome the performance gap caused by imperfect mention detection.

Summary updated

BibTeX
@inproceedings{lee2011stanford,
  author = {Heeyoung Lee and Yves Peirsman and Angel X. Chang and Nathanael Chambers and Mihai Surdeanu and Dan Jurafsky},
  title = {{Stanford's Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task.}},
  booktitle = {CoNLL Shared Task},
  year = {2011},
  url = {https://openreview.net/forum?id=IyIhJ9hp3e},
}
Figure