Supervised Coreference Resolution with SUCRE

Hamidreza Kobdani, Hinrich SchützeCoNLL Shared Task 2011 · OpenReview

Abstract

Hamidreza Kobdani, Hinrich Schuetze. Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task.

2011.

Introduction

Coreference resolution is the task of determining when different words or phrases (markables) in a document refer to the same entity. For example, in the sentences "The scientist finished her experiment. She was tired," a coreference system must recognize that "The scientist" and "She" are the same person.

This is a notoriously difficult problem in Natural Language Processing (NLP) because simple string matching is insufficient. Phrases like "the book" and "a book" share a headword but might refer to different things based on their definiteness. Similarly, "the students in Germany" and "the students in France" share many words but represent distinct groups. The authors address the need for a system that can handle these nuances through sophisticated feature engineering and a modular architecture.

Methodology

SUCRE (a name derived from its modular components) follows a standard two-step pipeline: classification and clustering.

Modular Architecture

The system maintains a clean separation between how data is stored, how features are defined, and which machine learning algorithm is used. Instead of hard-coding features, the authors introduced a "feature definition language." This allows them to access complex information, such as the semantic class of a word (via WordNet), its grammatical role, or specific words within a noun phrase (first, last, head).

Feature Selection and Learning

To avoid "noise," the authors used Information Gain (IG) and Information Gain Ratio (IGR). These metrics help determine which features actually help the model distinguish between coreferent and non-coreferent pairs. The core classifier is a Decision Tree (C4.5). While the authors tested Support Vector Machines (SVM) and Maximum Entropy models, they found that Decision Trees performed best for this specific setup. After classification, they use best-first clustering to group the links into final entity chains.

Experimental Setup

The system was evaluated using the OntoNotes corpus as part of the CoNLL-2011 Shared Task (Modeling Unrestricted Coreference in OntoNotes).

  • Training Set: 4,674 documents containing approximately 1.9 million tokens and 50,612 coreference chains.
  • Development Set: 303 documents used for tuning and reporting results.

Results and Discussion

The authors reported results across several standard metrics, including MUC, $B^3$, CEAF, and BLANC. A primary finding was the massive performance gap between using automatically detected markables versus "Gold" (human-annotated) markables.

Performance on the development set using automatic boundaries.

As shown in Figure 1, using automatic boundaries with automatic preprocessing yields a MUC $F_1$ score of 53.06. However, when gold boundaries are provided (Figure 2), this score jumps significantly to 70.34, even when other preprocessing remains automatic. This highlights that boundary detection is a major bottleneck for current systems.

Performance on the development set using gold boundaries.

Key Insights

  • Beyond the Headword: Relying solely on the "head word" of a phrase (e.g., "students") is often misleading. Features must account for modifiers and determiners to distinguish between similar-looking but distinct entities.
  • The Power of IGR: When a feature has many possible values (like a specific word), standard Information Gain can be biased. Using the Information Gain Ratio provides a more reliable indicator of a feature's true utility.
  • Semantic Grounding: Integrating WordNet to assign semantic classes (e.g., identifying "members of parliament" as "person" vs. "members of the EU" as "country") significantly aids the resolution process.
  • The "Singleton" Effect: The presence or absence of "singletons" (entities mentioned only once) can drastically swing evaluation metrics like $B^3$, emphasizing the importance of consistent preprocessing.

Conclusion

While the modular approach of SUCRE is flexible, the system's performance is heavily dependent on the quality of the initial preprocessing, particularly boundary detection. Despite the popularity of complex kernels in SVMs, the simpler Decision Tree approach remained the most effective, suggesting that the quality of the features extracted via the definition language was the primary driver of success. Future work involves refining the feature definition language to capture even more complex linguistic dependencies and exploring how to bridge the gap between automatic and gold-standard markable detection.

Summary updated

Abstract

Hamidreza Kobdani, Hinrich Schuetze. Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task. 2011.
BibTeX
@inproceedings{kobdani2011supervised,
  author = {Hamidreza Kobdani and Hinrich Schütze},
  title = {{Supervised Coreference Resolution with SUCRE}},
  booktitle = {CoNLL Shared Task},
  year = {2011},
  url = {https://openreview.net/forum?id=Dl4THBqSzkz},
}
Figure