Reconciling OntoNotes: Unrestricted Coreference Resolution in OntoNotes with Reconcile

Veselin Stoyanov, Uday Babbar, Pracheer Gupta, Claire CardieCoNLL Shared Task 2011 · OpenReview

Abstract

This paper describes the adaptation of the Reconcile coreference resolution platform for the 2011 CoNLL Shared Task, which utilized the extensive OntoNotes corpus. The primary challenge addressed is the $O(n^2)$ complexity of traditional all-pairs coreference models. The authors introduce the Smart Pair Generator (SMARTPG), a heuristic-based pruning method that reduces the search space for antecedents based on linguistic properties. The resulting system achieved an official combined score of 0.5192, ranking 8th out of 21 participants, and demonstrated that linear-time processing is achievable without significant loss in accuracy.

Introduction

Coreference resolution—identifying when different mentions in a text refer to the same entity—is a fundamental yet computationally demanding NLP task. As datasets scale to the size of OntoNotes, the standard approach of comparing every possible pair of mentions becomes a significant bottleneck. This work adapts the Reconcile platform to handle the specific requirements of the 2011 CoNLL Shared Task, focusing on balancing efficiency and performance.

The Reconcile Architecture and SMARTPG

The system employs a modular pipeline consisting of preprocessing, feature generation (61 features), classification via an averaged perceptron, single-link clustering, and scoring.

The core innovation is the Smart Pair Generator (SMARTPG). Instead of exhaustive comparison, SMARTPG applies linguistic heuristics to limit the antecedent search:

  • Proper Names: Compared to names within the last 20 sentences.
  • Pronouns: Compared to mentions within the last 2 sentences.
  • Definite NPs: Compared to mentions within the last 6 sentences.

This approach transforms the pairwise comparison from quadratic to linear complexity relative to the number of mentions. As shown in the comparison below, SMARTPG maintains performance levels nearly identical to the exhaustive "All Pairs" approach.

Comparison of SMARTPG and All Pairs generation performance.

Experimental Results

The system was trained on a subset of 750 OntoNotes documents due to memory constraints and evaluated on the official CoNLL 2011 test set. In the "Closed Task" category, the system achieved a combined score (averaging MUC, BCubed, and CEAF) of 0.5192.

The authors also evaluated the system using "Gold Mentions" to isolate the impact of mention detection. The score improved to 0.5355, indicating that while the coreference engine is robust, perfect mention extraction remains a significant area for potential gain.

Official CoNLL-2011 test scores for the closed task and gold mentions.

Key Insights and Discussion

A critical finding was the sensitivity of the system to the optimization metric. Thresholds optimized for BCubed did not necessarily yield the best results for MUC or CEAF. Furthermore, the authors noted that their internal scorers did not perfectly align with the official CoNLL scorer, which likely impacted their final ranking.

To ensure high coverage, the Coreference Element (CE) extractor was tuned for high recall (>96%). The underlying philosophy is that the downstream classifier can filter out false positives, but it cannot recover mentions missed during the initial extraction phase.

Conclusion and Future Work

The adaptation of Reconcile to OntoNotes demonstrated that heuristic pruning via SMARTPG is an effective strategy for large-scale coreference resolution. Future improvements include moving to an on-line perceptron to utilize the full training set, incorporating verb coreference, and aligning the training objective more closely with the official evaluation metrics.

Summary updated

BibTeX
@inproceedings{stoyanov2011reconciling,
  author = {Veselin Stoyanov and Uday Babbar and Pracheer Gupta and Claire Cardie},
  title = {{Reconciling OntoNotes: Unrestricted Coreference Resolution in OntoNotes with Reconcile}},
  booktitle = {CoNLL Shared Task},
  year = {2011},
  url = {https://openreview.net/forum?id=mRS_JYy23h9},
}
Figure