Abstract
Coreference resolution—the task of identifying when different expressions in a text refer to the same entity—has long been dominated by newswire datasets like OntoNotes. However, these datasets often represent an "easy mode" of language, characterized by sparse references and a lack of required world knowledge. We introduce a new, high-density coreference dataset based on Quiz Bowl trivia questions designed to challenge state-of-the-art models. By employing expert crowdsourcing and active learning, we demonstrate that training on complex, semantically rich data allows even simple models to outperform complex systems trained on traditional newswire.
The Newswire Monoculture
For years, progress in coreference resolution has been measured against datasets that are intentionally written to be easily understood by a general audience. Newswire text typically features sparse references and few nested mentions. This has created a "monoculture" where models excel at syntactic pattern matching but fail when faced with more expressive, complex text that requires deep world knowledge to decode. To "remove the training wheels," we developed a benchmark that is intentionally difficult, dense, and requires external knowledge to resolve.
The Quiz Bowl Dataset
Our new dataset consists of 400 documents containing 9,471 mentions. Unlike newswire, Quiz Bowl questions are "pyramidal," starting with obscure clues and ending with well-known facts. This structure creates an incredibly high concentration of coreferent terms (e.g., "this character," "this man," "he," "the protagonist") within a very short span.
As shown in the figure above, the Quiz Bowl dataset contains nearly double the mentions per sentence (3.7) compared to OntoNotes (2.1) and features a significantly higher rate of nested mentions—where one entity mention exists inside another.
Intelligent Annotation and Active Learning
To build this dataset efficiently, we utilized two primary strategies:
- Expert Crowdsourcing: We built a web-based tool for the Quiz Bowl community, allowing trivia enthusiasts to annotate questions as part of their practice. This provided high-quality data from users with the domain expertise required to identify obscure references.
- Active Learning: To maximize the utility of a small dataset, we employed "voting sampling." We trained multiple models on a seed set and selected new questions for annotation where the models showed the highest variance in disagreement.
Our results indicate that active learning significantly improved model precision compared to random sampling, allowing for a robust benchmark to be built with fewer total annotated documents.
Bridging the World Knowledge Gap
To prove the value of this data, we developed a relatively straightforward coreference system using a Conditional Random Field (CRF) for mention detection and a logistic regression classifier for pairwise coreference. Crucially, we incorporated word embeddings (word2vec and GloVe). These embeddings allowed the model to learn semantic proximities—for example, recognizing that "Russian" is closer to "Dostoevsky" than to "Hemingway."
This simple model, when trained on Quiz Bowl data, gained approximately 5 points in F1-score through the use of embeddings alone. Furthermore, it outperformed the complex, state-of-the-art Berkeley system (trained on newswire) across almost every metric (MUC, BCUB, CEAFE) when evaluated on our new dataset. This suggests that the nature of the training data is often more critical than the complexity of the algorithm.
Conclusion and Future Work
By moving beyond the simplicity of newswire, we have highlighted a significant gap in current NLP capabilities: the requirement for world knowledge. While our current dataset focuses heavily on literature, future work will expand into history and science. We have shown that when models are forced to "remove the training wheels" and engage with dense, expert-level text, the path toward true linguistic understanding becomes clearer.
Summary updated