They Exist! Introducing Plural Mentions to Coreference Resolution and Entity Linking

Ethan Zhou, Jinho D. ChoiCOLING 2018

TL;DR

This paper tackles the under-explored challenge of resolving plural mentions (e.g., "they," "we") in coreference resolution and entity linking tasks. They introduce a new annotated dataset and novel algorithms that significantly outperform existing models designed solely for singular mentions, paving the way for more robust NLP systems.

What Problem Does This Paper Address?

Coreference resolution and entity linking are fundamental NLP tasks that aim to identify mentions in text that refer to the same real-world entity. For example, in "John went to the store. He bought milk," "John" and "He" refer to the same entity. Similarly, entity linking connects these mentions to specific entries in a knowledge base. While current models excel at handling singular mentions (referring to one entity), they struggle with plural mentions (referring to multiple entities). This is a significant limitation because plural mentions are common in natural language, especially in dialogues. Existing datasets and models often overlook or inadequately handle these plural references, leading to incomplete or incorrect entity identification. This paper aims to bridge this gap by developing methods and resources specifically for plural mention resolution.

Methods

The authors introduce two main novelties:

  • Expanded Corpus and Annotation Scheme: They build upon the existing Character Mining project's corpus (derived from TV show transcripts) by significantly expanding its size and, crucially, introducing manual annotations for plural mentions. Their annotation scheme allows plural mentions to be linked to sets of entities, a departure from previous work that often forced them into single entity clusters.
  • Novel Algorithms:
    • Coreference Resolution: They propose a new coreference resolution algorithm that selectively creates clusters to accommodate both singular and plural mentions. This algorithm handles the complexity that plural mentions can belong to multiple clusters and that referent relations are not always transitive with plurals.
  • Entity Linking: They adapt an existing Agglomerative Convolutional Neural Network (ACNN) architecture for entity linking. Their modified model, a multi-task learning approach, is designed to jointly handle both singular and plural mentions, leveraging the coreference resolution output.

Datasets

  • Expanded Character Mining Corpus: The primary dataset used is an expansion of the Character Mining project's corpus, derived from TV show transcripts (specifically, the first four seasons of "Friends"). This corpus was expanded by approximately 2.5 times its original size and, most importantly, augmented with manual annotations for plural mentions. This new dataset is the first of its kind at this scale for plural mention resolution.

Key Insights and Ideas

  • Plural Mentions are Not Negligible: The paper highlights that plural mentions constitute a significant portion (around 9%) of their annotated dataset, underscoring the need for dedicated handling.
  • Limitations of Existing Coreference Schemes: The CoNLL'12 annotation scheme, for instance, forces plural mentions into single noun phrase clusters, losing valuable information about the individual entities they refer to.
  • Transitivity Breakdown with Plurals: The non-transitive nature of coreference with plural mentions is a key challenge. If "they" refers to "John and Mary," and "they" also refers to "Peter and Jane," it doesn't imply "John" and "Peter" are coreferent.
  • Need for Adjusted Evaluation Metrics: Standard coreference evaluation metrics (like B³, CEAF, BLANC) needed adaptation to properly evaluate systems that handle plural mentions referring to multiple entities.
  • Joint Modeling is Key: The success of their entity linking model hinges on jointly handling singular and plural mentions, suggesting that these tasks are intertwined.

Key Results

  • Coreference Resolution: Their proposed coreference resolution model significantly outperforms the baseline (CZC, a model trained on singular mentions) across various metrics, particularly CEAF, demonstrating its effectiveness in handling plural mentions without sacrificing performance on singular ones. For example, their model achieved a CEAF F1 score of 59.0±0.5, compared to the baseline's 55.4±0.6.
  • Entity Linking: The entity linking model also shows strong performance, achieving competitive micro-average F1 scores for both singular and plural mentions. While the micro-average F1 for plural mentions had higher standard deviations, the overall results indicate a robust ability to link plural mentions to their correct entities. For instance, their model achieved a micro-average F1 of 67.4±0.8 for all mentions, compared to the baseline's 66.2±0.4.

Limitations and Future Work

  • Inter-Annotator Agreement for Plurals: The paper notes that the inter-annotator agreement for plural mentions was about 20% lower than for singular mentions, indicating the inherent complexity and subjectivity of annotating plural references.
  • Ambiguous Entity Types: While their model handles ambiguous entity types (like "GENERAL" or "OTHER"), further analysis is needed to understand potential biases introduced by crowdsourcing instructions.
  • Future Work: The authors suggest improving the quality and size of the dataset, and exploring the extraction of global and external features to further enhance coreference and entity resolution for both singular and plural mentions. They also plan to investigate the variance in micro-average F1 scores for plural mentions.

Summary updated

BibTeX
@inproceedings{zhou2018they,
  author = {Ethan Zhou and Jinho D. Choi},
  title = {{They Exist! Introducing Plural Mentions to Coreference Resolution and Entity Linking}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure