PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution

Hong Chen, Zhenhua Fan, Hao Lu, Alan L. Yuille, Shu Rong

Introduction

Coreference resolution—the task of identifying all expressions that refer to the same entity in a text—is a fundamental challenge in Natural Language Processing (NLP). For years, the benchmark for this task has been the OntoNotes dataset. However, OntoNotes suffers from two significant limitations: a high rate of rare, out-of-vocabulary (OOV) words that conflate linguistic logic with vocabulary knowledge, and the exclusion of "singletons" (entities mentioned only once). The latter prevents models from being properly trained or evaluated on mention detection independently of clustering.

To address these gaps, researchers have introduced PreCo, a massive new dataset that is ten times larger than OntoNotes and specifically designed to isolate the logic of coreference from the complexities of advanced vocabulary.

The "Preschool" Advantage

PreCo is built from English reading comprehension tests designed for students in China. While the source is academic, the vocabulary is restricted to approximately 3,000 words—a level comparable to that of an English-speaking preschooler. This "closed-world" vocabulary ensures that models are likely to have encountered nearly every word during training, reducing the OOV rate from 2.1% in OntoNotes to just 0.8% in PreCo. By minimizing lexical distractions, researchers can more accurately measure a model's ability to process the underlying logic of entity relationships.

Methodology and Annotation

The construction of PreCo involved a rigorous multi-stage pipeline to ensure high data quality and comprehensive coverage. Unlike previous datasets, PreCo includes annotations for both coreferring mentions and singletons, providing a more complete picture of entity distribution in natural language.

The multi-stage data collection and annotation pipeline used to create the PreCo dataset, from crawling to final ensemble annotation.

The annotation process utilized a tiered approach:

  1. Independent Labeling: Three separate annotators labeled each document independently.
  2. Merging: A second tier of annotators reviewed and merged these results to resolve discrepancies.
  3. Ensemble: A final algorithmic ensemble was used to produce the "gold standard" annotations.

Key Results and Insights

PreCo consists of 38,000 documents and 12.5 million words, making it one of the largest resources of its kind. Experiments using state-of-the-art models like EE2E-Coref revealed several critical insights into the current state of NLP.

The Impact of Scale and Vocabulary

As shown in the experimental results, increasing the training set size directly correlates with improved F1 scores and a significant reduction in the OOV rate. This validates the authors' hypothesis that a large-scale, low-vocabulary dataset provides a cleaner environment for error analysis.

Performance metrics (Avg. F1, OOV rate, and LFW rate) plotted against training set size and subset indices, demonstrating the benefits of PreCo's scale.

Clustering as the Primary Bottleneck

By including singletons, researchers were able to conduct "perfect mention" experiments. When the model was provided with ground-truth mentions and tasked only with clustering them, a significant performance gap remained. This suggests that the primary difficulty in coreference resolution is not finding nouns, but rather representing the complex relationships between entities across multiple sentences.

The Human-AI Gap

While models achieved an F1 score of 81.5 on PreCo, human performance reached 87.9. This 6-point gap indicates that even with a simplified vocabulary, the core logic of coreference remains a non-trivial challenge for artificial intelligence, particularly when context spans across a long document.

Conclusion

PreCo represents a significant shift in coreference resolution research. By providing a massive, singleton-aware dataset with a controlled vocabulary, it allows the community to move past the "vocabulary problem" and focus on the "logic problem." Future work will likely leverage PreCo to improve entity representation and multi-sentence contextual understanding, pushing AI closer to human-level reading comprehension.

Summary updated

Abstract

We introduce PreCo, a large-scale English dataset for coreference resolution. The dataset is designed to embody the core challenges in coreference, such as entity representation, by alleviating the challenge of low overlap between training and test sets and enabling separated analysis of mention detection and mention clustering. To strengthen the training-test overlap, we collect a large corpus of about 38K documents and 12.4M words which are mostly from the vocabulary of English-speaking preschoolers. Experiments show that with higher training-test overlap, error analysis on PreCo is more efficient than the one on OntoNotes, a popular existing dataset. Furthermore, we annotate singleton mentions making it possible for the first time to quantify the influence that a mention detector makes on coreference resolution performance. The dataset is freely available at this https URL
BibTeX
@inproceedings{chen2018preco,
  author = {Hong Chen and Zhenhua Fan and Hao Lu and Alan L. Yuille and Shu Rong},
  title = {{PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1016},
}
Figure