Abstract
Stanford’s winning entry for the CoNLL-2011 coreference resolution task demonstrates that a "sieve" of deterministic, rule-based models—applied in order from highest to lowest precision—can outperform complex machine learning approaches. By propagating information like gender and number across entity clusters, the system makes increasingly informed decisions as it moves from simple string matching to difficult pronoun resolution.
Introduction
Coreference resolution is the task of identifying all expressions (mentions) in a text that refer to the same real-world entity. This task is notoriously difficult because it requires a mix of syntactic rules, semantic knowledge (e.g., knowing that a "plane" is an "aircraft"), and discourse understanding. Many systems struggle to balance precision (avoiding incorrect links) and recall (finding all correct links). The authors aim to build a robust pipeline that handles these diverse challenges without the need for complex, often opaque, global optimization models.
Methodology
The Multi-Pass Sieve Architecture
The core of the system is the Multi-Pass Sieve. Instead of making all coreference decisions at once, the system applies 13 distinct "sieves" in a specific order, starting with very "safe" rules such as Exact String Match. Once two mentions are linked into a cluster, their attributes (like gender, number, and animacy) are shared across the entire cluster. This Attribute Propagation is a key innovation: if one mention in a cluster is identified as "singular" and "male," every other mention in that cluster inherits those traits, helping subsequent, "riskier" sieves (like pronoun resolution) avoid mistakes.
Discourse and Semantic Extensions
The system incorporates specialized sieves to handle complex linguistic phenomena:
- Discourse Sieve: Uses speaker information to resolve pronouns. By identifying the speaker of a quote, the system can confidently link "I" to the speaker and "you" to the listener.
- Semantic Sieves: Utilizes external knowledge bases like WordNet, Wikipedia, and Freebase to identify aliases (e.g., "America Online" and "AOL") or lexical chains (e.g., "Britain" and "country").
Experimental Evaluation
The system was evaluated on the CoNLL-2011 Shared Task dataset (OntoNotes corpus), covering news, magazine articles, and broadcast conversations. Evaluation was split into a Closed Track (limited to provided data) and an Open Track (allowing external knowledge sources).
Performance Analysis
An ablation study highlights the impact of different components. The discourse sieve provided a significant boost (nearly 2 points), while semantic sieves showed mixed results, sometimes introducing noise by linking incorrectly detected mentions.
Results
Stanford’s system ranked first in both the open and closed tracks of the CoNLL-2011 Shared Task. The official results demonstrate the system's robustness across different mention boundary conditions.
Discussion
The Mention Detection Bottleneck
A critical finding was the massive performance gap—approximately 15 points—between using "gold" (human-annotated) mentions versus "predicted" (machine-generated) mentions. This suggests that a major hurdle in coreference resolution is not just the linking logic, but the initial identification of valid mentions.
Simplicity and Precision
Despite the trend toward complex joint-inference models, this work proves that a well-engineered pipeline of simple deterministic rules can achieve state-of-the-art results. By prioritizing precision-first architecture, the system builds a reliable foundation of entity clusters that reduces the search space for harder cases later in the pipeline.
Conclusion
The Multi-Pass Sieve system demonstrates that modularity and attribute propagation are powerful tools for coreference resolution. Future work may focus on "joint modeling" of mentions and coreference chains to overcome the performance gap caused by imperfect mention detection.
Summary updated