Introduction
Coreference resolution—the task of identifying all expressions that refer to the same entity in a text—is a fundamental challenge in Natural Language Processing (NLP). For years, the benchmark for this task has been the OntoNotes dataset. However, OntoNotes suffers from two significant limitations: a high rate of rare, out-of-vocabulary (OOV) words that conflate linguistic logic with vocabulary knowledge, and the exclusion of "singletons" (entities mentioned only once). The latter prevents models from being properly trained or evaluated on mention detection independently of clustering.
To address these gaps, researchers have introduced PreCo, a massive new dataset that is ten times larger than OntoNotes and specifically designed to isolate the logic of coreference from the complexities of advanced vocabulary.
The "Preschool" Advantage
PreCo is built from English reading comprehension tests designed for students in China. While the source is academic, the vocabulary is restricted to approximately 3,000 words—a level comparable to that of an English-speaking preschooler. This "closed-world" vocabulary ensures that models are likely to have encountered nearly every word during training, reducing the OOV rate from 2.1% in OntoNotes to just 0.8% in PreCo. By minimizing lexical distractions, researchers can more accurately measure a model's ability to process the underlying logic of entity relationships.
Methodology and Annotation
The construction of PreCo involved a rigorous multi-stage pipeline to ensure high data quality and comprehensive coverage. Unlike previous datasets, PreCo includes annotations for both coreferring mentions and singletons, providing a more complete picture of entity distribution in natural language.
The annotation process utilized a tiered approach:
- Independent Labeling: Three separate annotators labeled each document independently.
- Merging: A second tier of annotators reviewed and merged these results to resolve discrepancies.
- Ensemble: A final algorithmic ensemble was used to produce the "gold standard" annotations.
Key Results and Insights
PreCo consists of 38,000 documents and 12.5 million words, making it one of the largest resources of its kind. Experiments using state-of-the-art models like EE2E-Coref revealed several critical insights into the current state of NLP.
The Impact of Scale and Vocabulary
As shown in the experimental results, increasing the training set size directly correlates with improved F1 scores and a significant reduction in the OOV rate. This validates the authors' hypothesis that a large-scale, low-vocabulary dataset provides a cleaner environment for error analysis.
Clustering as the Primary Bottleneck
By including singletons, researchers were able to conduct "perfect mention" experiments. When the model was provided with ground-truth mentions and tasked only with clustering them, a significant performance gap remained. This suggests that the primary difficulty in coreference resolution is not finding nouns, but rather representing the complex relationships between entities across multiple sentences.
The Human-AI Gap
While models achieved an F1 score of 81.5 on PreCo, human performance reached 87.9. This 6-point gap indicates that even with a simplified vocabulary, the core logic of coreference remains a non-trivial challenge for artificial intelligence, particularly when context spans across a long document.
Conclusion
PreCo represents a significant shift in coreference resolution research. By providing a massive, singleton-aware dataset with a controlled vocabulary, it allows the community to move past the "vocabulary problem" and focus on the "logic problem." Future work will likely leverage PreCo to improve entity representation and multi-sentence contextual understanding, pushing AI closer to human-level reading comprehension.
Summary updated