TL;DR
State-of-the-art coreference resolution models, despite impressive performance on the OntoNotes benchmark, show a significant performance drop when evaluated on new, diverse genres. This paper introduces OntoGUM, a new large-scale dataset covering 12 genres, and demonstrates that even advanced contextualized language models struggle with generalization, highlighting a critical overfitting problem.
What Problem Does This Paper Address?
Coreference resolution, the task of identifying mentions that refer to the same entity (e.g., "Barack Obama" and "he"), is crucial for natural language understanding. While models have achieved high scores on the OntoNotes benchmark, there's a growing concern that these systems might be "overfitting" to the specific characteristics of OntoNotes data. This means they might not generalize well to text from different domains or genres. Previous attempts to evaluate this generalization have been limited by small datasets or a narrow focus on specific types of coreference. The authors aim to address this by creating a more comprehensive out-of-domain evaluation benchmark and demonstrating the extent of the generalization problem.
Methods
The core of the paper involves creating a new dataset, OntoGUM, by converting the existing Georgetown University Multilayer (GUM) corpus. This conversion process used deterministic rules to align GUM's annotations with the OntoNotes coreference scheme. They then evaluated two types of coreference resolution systems on this new dataset:
- Deterministic Coreference Model (dcoref): A rule-based system from Stanford CoreNLP. This serves as a baseline and is expected to be more stable across genres due to its reliance on linguistic rules rather than learned patterns from specific data.
- State-of-the-Art (SOTA) Neural Model: An end-to-end system that leverages pretrained contextualized language models (specifically, SpanBERT-large fine-tuned on OntoNotes). This represents the current best performance on OntoNotes.
The evaluation focused on comparing the performance of these systems on OntoNotes versus OntoGUM, and also on a genre-by-genre basis within OntoGUM.
Datasets
- OntoNotes: The standard benchmark for coreference resolution, used for training and evaluating existing SOTA models.
- GUM (Georgetown University Multilayer Corpus): An existing English corpus with rich annotations across 12 genres.
- OntoGUM: The newly introduced dataset, created by converting GUM to follow the OntoNotes coreference annotation scheme. It is described as the largest open, gold coreference dataset consistent with OntoNotes, featuring 168 documents, approximately 150K tokens, and over 19,000 mentions across 12 genres.
Key Insights and Ideas
- Significant Generalization Gap: The most striking finding is the substantial performance degradation of SOTA models when moving from OntoNotes to OntoGUM. This strongly suggests that current models are indeed overfitting to OntoNotes.
- Contextualized Models Aren't Immune: Even advanced models using contextualized language embeddings like BERT (and SpanBERT) are not immune to this overfitting problem, showing a notable drop in performance.
- Genre Matters: Performance varies significantly across the 12 genres in OntoGUM. Genres with more pronouns and gold speaker information tend to yield higher scores, while those with fewer pronouns and less speaker information are more challenging.
- Lexical Overlap is Key: The paper reinforces the idea that high lexical overlap between training and test data (as seen in OntoNotes) contributes to inflated performance, masking true generalization capabilities.
Key Results
- The SOTA neural model (e2e+SpanBERT) experienced a 15-20% degradation in performance on OntoGUM compared to OntoNotes.
- The deterministic model (dcoref) also saw a significant drop, with over 18 points (30%) lower scores on OntoGUM.
- Performance varied widely across genres, with some genres showing scores as low as the 30s and others reaching into the 70s for the SOTA model.
- The paper highlights that even genres that are somewhat similar to OntoNotes (like news) can still pose challenges, and genres with distinct characteristics (like spontaneous speech vlogs) can be particularly difficult.
Limitations and Future Work
- Mention Detection Bias: The authors acknowledge that input data for the deterministic system (predicted by CoreNLP) might have a domain bias favoring newswire, potentially contributing to the observed degradation.
- OntoGUM Size: While OntoGUM is the largest new dataset of its kind, it is still smaller than the OntoNotes coreference dataset, which might limit training SOTA models directly on it while maintaining peak performance.
- Future Work: The paper positions OntoGUM as a valuable benchmark for future research, encouraging the development of more robust and generalizable coreference resolution systems that can perform well "in the wild." Further investigation into why certain genres are more difficult and how to improve performance on them is a clear direction.
Summary updated