A Fine-grained Large-scale Analysis of Coreference Projection

Michal NovákNAACL 2018

TL;DR

This paper presents a comprehensive analysis of coreference projection, a technique for transferring coreference annotations between languages, using a large-scale Czech-English parallel corpus. The authors introduce a novel factorized analysis that breaks down projection errors by mention type and contributing factors, revealing that alignment and mention matching are the primary bottlenecks, particularly for nominal groups and named entities.

What Problem Does This Paper Address?

Coreference resolution, the task of identifying mentions in text that refer to the same real-world entity, is crucial for many Natural Language Processing (NLP) tasks. However, creating high-quality coreference annotations is labor-intensive and expensive, especially for low-resource languages. Coreference projection offers a way to leverage existing annotations from resource-rich languages to build coreference systems for other languages. While projection has shown promise, its performance has often been mediocre, and a detailed understanding of why it fails has been lacking. This paper addresses this gap by conducting a fine-grained, large-scale analysis of coreference projection, aiming to pinpoint the specific challenges and identify areas for improvement.

Methods

The authors employ a corpus-based projection approach, utilizing a human-translated parallel corpus. Their method projects "gold" (manually annotated) coreference links from one language to another. A key innovation is their factorized analysis, which dissects projection errors by:

  • Mention Type: Analyzing performance across various categories of mentions (pronouns, named entities, nominal groups, zero anaphora, etc.).
  • Contributing Factors: Attributing errors to specific stages of the projection pipeline:
  • Alignment: The quality of word-level alignment between the parallel sentences.
  • Mention Matching: Whether corresponding mentions in the target language are correctly identified.
  • Antecedent Selection: Whether the correct antecedent for a projected anaphor is found.

This detailed breakdown allows for a deeper understanding of where projection falters. They also specifically address zero anaphora, which are unexpressed arguments (common in pro-drop languages like Czech) and are often overlooked in previous projection work.

Datasets

The primary dataset used is the Prague Czech-English Dependency Treebank 2.0 Coref (PCEDT 2.0 Coref). This is a large, manually annotated parallel corpus of Czech and English sentences, featuring coreference annotations at the tectogrammatical layer. The authors note that they use a slightly modified version with supervised alignment for their experiments.

Key Insights and Ideas

  • Projection is Hard: Even with gold annotations, coreference projection between Czech and English achieves only around 50% F-score, suggesting inherent difficulties in cross-lingual coreference transfer.
  • Alignment and Mention Matching are Bottlenecks: The analysis strongly indicates that errors in word alignment and the inability to correctly match mentions in the target language are the primary drivers of projection failures, particularly impacting recall.
  • Pronouns Project Better than Nominal Groups: Similar to previous work, pronouns exhibit higher projection accuracy than nominal groups and named entities. This is attributed to their more consistent and often shorter spans, making them easier to align and match.
  • Zero Anaphora are Important and Addressable: The paper highlights the significance of zero anaphora, especially in pro-drop languages like Czech, and demonstrates that their tectogrammatical representation allows for their inclusion in projection analysis.
  • Language-Specific Challenges: The study reveals differences in projection performance between English-to-Czech (EN→CS) and Czech-to-English (CS→EN) directions, with CS→EN being slightly easier but still challenging.

Key Results

  • Overall F-scores: The overall F-scores for coreference projection are around 50% for both EN→CS and CS→EN directions.
  • Recall is the Main Issue: While precision is not perfect, recall rates are significantly lower, indicating that many potential coreference links are simply not projected.
  • Mention Type Performance: Pronouns and zero subjects generally show high scores (often near 100% for mention matching), while nominal groups and named entities exhibit lower scores, especially in precision.
  • Czech Reflexive Pronouns are Problematic: These show particularly low alignment rates, often due to linguistic differences where a reflexive pronoun in Czech is not explicitly expressed or is expressed differently in English.

Limitations and Future Work

  • Gold Annotations: The experiments are conducted on gold coreference annotations. The authors acknowledge that performance with automatically generated annotations would likely be worse.
  • Domain Specificity: The dataset (Wall Street Journal) might influence the results, and findings may not generalize to other domains.
  • Czech Demonstrative Pronouns: These show surprisingly low scores, with errors attributed to annotator issues, alignment problems, and the inherently flexible nature of demonstratives that can refer to various linguistic elements. Further investigation on diverse data is suggested.
  • Span Boundaries: While the tectogrammatical representation doesn't explicitly define mention spans, the authors note that confirming their hypothesis about shorter mentions being easier to project would benefit from defining and analyzing span boundaries.
  • Need for Broader Verification: The unaligned nature of Czech personal pronouns requires verification on data from various domains and translation directions.

Summary updated

BibTeX
@inproceedings{novak2018fine,
  author = {Michal Novák},
  title = {{A Fine-grained Large-scale Analysis of Coreference Projection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure