A Fully Coreference-annotated Corpus of Scholarly Papers from the ACL Anthology

Ulrich Schäfer, Christian Spurk, Jörg SteffenCOLING 2012

TL;DR

This paper introduces a large, manually annotated corpus of coreference relations within scientific papers from the ACL Anthology. This resource addresses the gap in coreference resolution research, which has historically focused on news text, by providing a domain-specific dataset that can improve NLP applications in scientific literature.

What Problem Does This Paper Address?

Coreference resolution (CR) is a fundamental task in Natural Language Processing (NLP) that involves identifying all mentions of real-world entities in a text. For example, recognizing that "the director," "he," and "John Smith" all refer to the same person. While CR has been extensively studied, most research and available annotated corpora (like MUC and ACE) have focused on news text. This poses a problem because scientific text, the domain of this paper, has distinct characteristics that make coreference resolution more challenging and potentially require different approaches. Scientific texts often feature more complex sentence structures, a higher proportion of definite descriptions, and a greater emphasis on domain-specific terminology and abstract entities rather than just person names. Systems trained solely on news text may perform poorly on scientific documents, highlighting the need for domain-specific resources.

Methods

The core of this paper is the creation of a new, large-scale coreference-annotated corpus. The process involved:

  • Corpus Selection: 266 scientific papers were selected from the ACL Anthology, specifically from the P08 (ACL-2008 long papers), D07 (EMNLP-CoNLL-2007), and C02 (COLING 2002) sections.
  • Text Extraction: Papers were converted from PDF to text files using a commercial OCR program to ensure a uniform, though not perfect, quality, and to be independent of PDF generator variations.
  • Manual Annotation: Human annotators, advanced students of English language and literature with general linguistics knowledge, performed the coreference annotation. They were tasked with identifying all mentions of entities and grouping coreferential mentions into sets. The annotation process was designed to be comprehensive, covering the full textual content of each paper.
  • Annotation Schema: The annotation schema was adapted from the ACE program's "entity detection and tracking" task. Notably, unlike ACE, this corpus does not restrict the types of entities to be annotated, aiming for a more comprehensive coverage of scientific discourse. Annotators identified maximal mention extents, including all modifiers. Coreference sets were created for mentions that were coreferential with at least one other mention in the text, with an emphasis on maximal coreference sets.
  • Two-Phase Annotation: The annotation was performed in two phases: an initial annotation and a secondary correction phase. This approach was used to measure inter-annotator agreement and improve the quality and consistency of the annotations.
  • Automatic Annotations: To provide negative examples for machine learning systems (mentions not coreferential with others), automatic annotations were generated for NPs, possessive determiners, and proper names that were not part of any entity mention. This was done using the Stanford Parser and the SProUT system.

Datasets

  • ACL Anthology Corpus: The primary dataset used for annotation is a subset of 266 scientific papers from the ACL Anthology (sections P08, D07, and C02).
  • Comparison to Existing Corpora: The paper explicitly compares the size of their newly created corpus to existing coreference corpora like ACE. The ACL Anthology corpus is described as "by far larger" than the ACE coreference corpora. For instance, the manually annotated corpus contains 1,326,147 tokens and 48,960 sentences, significantly exceeding ACE 2004's 189,620 tokens and 5,654 sentences.

Key Insights and Ideas

  • Domain Shift Matters: The paper strongly suggests that coreference resolution systems trained on news text perform worse on scientific text. This highlights the critical need for domain-specific annotated resources to build robust NLP models for scientific literature.
  • Scientific Text Complexity: Scientific texts present unique challenges for coreference resolution, including a higher prevalence of definite descriptions and references to abstract entities and domain terminology, which differ from the person- and organization-centric nature of news text.
  • Comprehensive Annotation: By not restricting entity types, the authors aimed to capture a broader range of coreferential phenomena relevant to scientific discourse, which could be beneficial for tasks beyond traditional entity tracking.
  • Two-Stage Annotation for Quality: The use of an initial annotation followed by a correction phase, and later a single corrector for the entire corpus, demonstrates a commitment to achieving high-quality and consistent annotations, crucial for training reliable NLP models.

Key Results

  • Corpus Size: The manually annotated corpus contains approximately 1.3 million tokens and 48,960 sentences, making it a substantial resource for coreference research.
  • Inter-Annotator Agreement: In the initial annotation phase, an inter-annotator agreement of 49.5 MUC points was achieved. While acknowledged as low and prompting improvements, this metric provides a baseline for the difficulty of the task in this domain.
  • Annotation Effort: The second round of annotation and correction, performed by a single person, took approximately 9 months of part-time work (8-10 hours/week), underscoring the labor-intensive nature of creating high-quality coreference annotations.
  • Mention Types: The corpus includes various mention types, with definite noun phrases (def-np) being the most frequent (32,547), followed by proper names (ne) (14,451) and personal pronouns (pper) (5,921).

Limitations and Future Work

  • Imperfect Corpus: The authors acknowledge that the corpus, in its current state, is not perfect and may require further rounds of annotation assessment and correction.
  • Inter-Annotator Agreement: The initial inter-annotator agreement was found to be low, indicating the need for improved annotation guidelines and processes, which were subsequently implemented.
  • Future Work: The paper suggests that the corpus can be used to train coreference resolution systems that can, in turn, improve other NLP tasks such as semantic search, taxonomy extraction, question answering, citation analysis, and scientific discourse analysis. The release of the corpus is intended to encourage the scientific community to further improve it.

Summary updated

BibTeX
@inproceedings{schafer2012fully,
  author = {Ulrich Schäfer and Christian Spurk and Jörg Steffen},
  title = {{A Fully Coreference-annotated Corpus of Scholarly Papers from the ACL Anthology}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure