Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers

Anupam Guha, Mohit Iyyer, Danny Bouman, Jordan L. Boyd-Graber

Abstract

Coreference resolution—the task of identifying when different expressions in a text refer to the same entity—has long been dominated by newswire datasets like OntoNotes. However, these datasets often represent an "easy mode" of language, characterized by sparse references and a lack of required world knowledge. We introduce a new, high-density coreference dataset based on Quiz Bowl trivia questions designed to challenge state-of-the-art models. By employing expert crowdsourcing and active learning, we demonstrate that training on complex, semantically rich data allows even simple models to outperform complex systems trained on traditional newswire.

The Newswire Monoculture

For years, progress in coreference resolution has been measured against datasets that are intentionally written to be easily understood by a general audience. Newswire text typically features sparse references and few nested mentions. This has created a "monoculture" where models excel at syntactic pattern matching but fail when faced with more expressive, complex text that requires deep world knowledge to decode. To "remove the training wheels," we developed a benchmark that is intentionally difficult, dense, and requires external knowledge to resolve.

The Quiz Bowl Dataset

Our new dataset consists of 400 documents containing 9,471 mentions. Unlike newswire, Quiz Bowl questions are "pyramidal," starting with obscure clues and ending with well-known facts. This structure creates an incredibly high concentration of coreferent terms (e.g., "this character," "this man," "he," "the protagonist") within a very short span.

Comparison of mention and nested coreference density between Newswire and Quiz Bowl datasets.

As shown in the figure above, the Quiz Bowl dataset contains nearly double the mentions per sentence (3.7) compared to OntoNotes (2.1) and features a significantly higher rate of nested mentions—where one entity mention exists inside another.

Intelligent Annotation and Active Learning

To build this dataset efficiently, we utilized two primary strategies:

  1. Expert Crowdsourcing: We built a web-based tool for the Quiz Bowl community, allowing trivia enthusiasts to annotate questions as part of their practice. This provided high-quality data from users with the domain expertise required to identify obscure references.
  2. Active Learning: To maximize the utility of a small dataset, we employed "voting sampling." We trained multiple models on a seed set and selected new questions for annotation where the models showed the highest variance in disagreement.
Precision gains using active learning (voting sampling) versus random sampling across iterations.

Our results indicate that active learning significantly improved model precision compared to random sampling, allowing for a robust benchmark to be built with fewer total annotated documents.

Bridging the World Knowledge Gap

To prove the value of this data, we developed a relatively straightforward coreference system using a Conditional Random Field (CRF) for mention detection and a logistic regression classifier for pairwise coreference. Crucially, we incorporated word embeddings (word2vec and GloVe). These embeddings allowed the model to learn semantic proximities—for example, recognizing that "Russian" is closer to "Dostoevsky" than to "Hemingway."

This simple model, when trained on Quiz Bowl data, gained approximately 5 points in F1-score through the use of embeddings alone. Furthermore, it outperformed the complex, state-of-the-art Berkeley system (trained on newswire) across almost every metric (MUC, BCUB, CEAFE) when evaluated on our new dataset. This suggests that the nature of the training data is often more critical than the complexity of the algorithm.

Conclusion and Future Work

By moving beyond the simplicity of newswire, we have highlighted a significant gap in current NLP capabilities: the requirement for world knowledge. While our current dataset focuses heavily on literature, future work will expand into history and science. We have shown that when models are forced to "remove the training wheels" and engage with dense, expert-level text, the path toward true linguistic understanding becomes clearer.

Summary updated

Abstract

Coreference is a core nlp problem. However, newswire data, the primary source of existing coreference data, lack the richness necessary to truly solve coreference. We present a new domain with denser references—quiz bowl questions—that is challenging and enjoyable to humans, and we use the quiz bowl community to develop a new coreference dataset, together with an annotation framework that can tag any text data with coreferences and named entities. We also successfully integrate active learning into this annotation pipeline to collect documents maximally useful to coreference models. State-of-the-art coreference systems underperform a simple classifier on our new dataset, motivating non-newswire data for future coreference research.
BibTeX
@inproceedings{guha2015removing,
  author = {Anupam Guha and Mohit Iyyer and Danny Bouman and Jordan L. Boyd-Graber},
  title = {{Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1117},
}
Figure