Introduction
Coreference resolution—the task of identifying all expressions that refer to the same entity in a text—is a fundamental component of natural language processing (NLP). Despite its importance, progress has been hindered by two primary issues in existing benchmarks like OntoNotes: vocabulary sparsity and the exclusion of "singletons" (mentions that appear only once). When a model fails, it is often unclear if the failure is due to flawed coreference logic or simply an inability to recognize rare words.
To address these challenges, researchers have introduced PreCo, a massive new dataset that is ten times larger than OntoNotes. By restricting the vocabulary to a "preschool" level and annotating every mention, PreCo allows researchers to isolate the core logic of entity linking from lexical distractions.
Methodology: A Multi-Stage Annotation Pipeline
The authors constructed PreCo by crawling English reading comprehension tests designed for students in China. While the source material is for older students, the vocabulary (roughly 3,000 words) matches that of an English-speaking preschooler, providing a "clean" environment for testing relational logic. To ensure high-quality data, a rigorous multi-stage annotation pipeline was implemented.
As shown in the pipeline, each document underwent three independent annotation passes. These were then reconciled by three "merger" annotators who resolved discrepancies. Finally, an ensemble algorithm merged these results to establish a "gold standard" ground truth. This process ensures that the resulting 38,000 documents are both accurate and consistent.
Key Insights: The Singleton Mystery and Clustering Bottleneck
A unique feature of PreCo is its inclusion of singleton mentions, which constitute approximately 50.8% of the dataset. Most existing datasets ignore these, making it difficult to evaluate the first half of a coreference system: the mention detector.
By providing singleton data, the authors conducted a revealing experiment: they provided their model with "perfect" mention detection (an oracle) and observed only a modest jump in F1 score (from 77.3 to 81.6). This suggests a major insight for the field: the primary bottleneck in coreference resolution is the clustering logic (linking mentions), not the initial detection of mentions.
Experimental Results and Vocabulary Analysis
PreCo's scale and restricted vocabulary significantly reduce the "out-of-vocabulary" (OOV) rate, which stands at just 0.8% compared to 2.1% in OntoNotes. This low OOV rate makes error analysis more efficient, as researchers can focus on structural model failures rather than data scarcity issues.
As illustrated in the performance metrics, increasing the training set size consistently lowers both the OOV and Low-Frequency Word (LFW) rates. Using the state-of-the-art EE2E-Coref model, the authors achieved an average F1 score of 81.5 on PreCo. While this is a strong result, it remains below the human performance level of 87.9, representing the "unsolved" logic of entity representation that PreCo is designed to help bridge.
Conclusion
PreCo represents a significant shift in coreference resolution research by providing a massive, low-noise environment for model development. While it is not a general-purpose dataset for complex domains like law or medicine, it provides the necessary scale and detail to refine the fundamental mechanisms of mention clustering. Future work will likely focus on improving how models represent entities across multiple sentences, leveraging the singleton data provided by this new benchmark.
Summary updated