Search Space Pruning: A Simple Solution for Better Coreference Resolvers

Nafise Sadat Moosavi, Michael StrubeNAACL 2016

TL;DR

This paper tackles the significant performance gap in coreference resolution between systems using gold-standard mentions versus system-generated mentions. They propose a simple yet effective "singleton detection" method to prune the search space, achieving state-of-the-art results on the CoNLL 2012 English test set. This work highlights that simplifying the search space is a crucial, often overlooked, avenue for improving coreference resolution systems.

What Problem Does This Paper Address?

Coreference resolution is the task of identifying all mentions in a text that refer to the same real-world entity and grouping them into clusters. This is a fundamental but challenging problem in Natural Language Processing (NLP). A major hurdle is the sheer size and complexity of the "search space" – all possible ways to group mentions. When systems rely on their own predicted mentions (system mentions) rather than perfect, human-annotated ones (gold mentions), this search space becomes even larger and more unbalanced, leading to a significant drop in performance. The paper highlights a substantial 17% gap between gold and system mention performance on the CoNLL 2012 English test set, indicating that improving how systems navigate this search space is critical.

Methods

The core contribution of this paper is a singleton detection method. A "singleton" is a mention that refers to an entity that appears only once in the discourse. Identifying these non-coreferent mentions early can significantly reduce the search space for coreference resolvers.

The authors propose a simple yet effective singleton detection model using a small set of shallow features. These features include:

  • Lemmas of words within the mention and surrounding words.
  • Part-of-speech tags of words within the mention and surrounding words.
  • The mention string itself and its length.
  • Mention type (proper, nominal, pronominal).
  • Whether the whole mention string or its head word appears elsewhere in the document.

These features are fed into an anchored Support Vector Machine (SVM) with a polynomial kernel. They evaluate this model in three configurations:

  1. Surface: Uses only the shallow features described above.
  2. Combined: Adds linguistic features from prior work (Marneffe et al., 2015).
  3. Confident: A high-precision variant of the Surface configuration, designed to avoid over-pruning.

Datasets

  • CoNLL 2012 English development set: Used for intrinsic evaluation of the singleton detection model and for comparing with prior work (Marneffe et al., 2015).
  • CoNLL 2012 English training set: Used to train the singleton detection model.
  • CoNLL 2012 English test set: Used for extrinsic evaluation of the singleton detection model integrated into coreference resolution systems.

The paper does not introduce new datasets but evaluates on existing, standard benchmarks for coreference resolution.

Key Insights and Ideas

  • Search space pruning is a simple but powerful lever: The paper convincingly demonstrates that focusing on reducing the search space, rather than solely on more complex features or models, can yield significant improvements.
  • Singleton detection is more effective than anaphoricity detection for pruning: While anaphoricity detection (identifying mentions that refer back to something) is common, singleton detection (identifying mentions that don't refer to anything else) is shown to be a more direct way to prune the search space for both anaphors and antecedents.
  • Shallow features can be surprisingly effective: The authors achieve state-of-the-art results in singleton detection using only a small set of relatively simple, surface-level features, challenging the notion that highly complex linguistic features are always necessary.
  • High precision is crucial for pruning: The "Confident" configuration, which prioritizes precision over recall, is key to effectively pruning the search space without discarding too many actual coreferent mentions.

Key Results

  • State-of-the-art singleton detection: The proposed singleton detection model, using only shallow features, achieves new state-of-the-art results on the CoNLL 2012 English development set, significantly outperforming prior work.
  • Improved coreference resolution: Integrating their singleton detection model as a preprocessing step significantly improves the performance of baseline coreference resolution systems:
  • Stanford rule-based system: ~2% improvement.
  • Cort pairwise model: ~1% improvement.
  • Cort latent ranking model: ~0.5% improvement.
  • Best reported overall score: The integration with the Cort latent ranking model achieved the best reported overall score on the CoNLL 2012 English test set at the time of publication.
  • Statistical significance: All improvements reported on the test set were statistically significant (p < 0.05).

Limitations and Future Work

  • Limitations of current features: The paper acknowledges that detecting first mentions of entities is challenging. Features (8) and (9) (checking for string or head word matches) only address this to a limited degree. More context-aware features are needed.
  • Potential for over-pruning: While the "Confident" configuration aims for high precision, there's always a risk of over-pruning, especially with the latent ranking model where adding the singleton feature as a new input (rather than a preprocessing step) was necessary to avoid hurting recall.
  • Future work: The authors suggest that further improvements in singleton detection are still possible, especially by incorporating more context and features that consider other mentions. They also reiterate that search space pruning remains a promising direction for advancing coreference resolution. The singleton detection module is made available with the Cort source code.

Summary updated

BibTeX
@inproceedings{moosavi2016search,
  author = {Nafise Sadat Moosavi and Michael Strube},
  title = {{Search Space Pruning: A Simple Solution for Better Coreference Resolvers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
}
Figure