Abstract
Standard coreference resolution systems, typically trained on formal news corpora like OntoNotes, struggle with the informal and noisy nature of Twitter. This paper investigates domain adaptation strategies for the e2e-coref system. By strategically retraining models on a combination of Twitter-specific data and spoken language genres, we demonstrate a significant performance boost. Our results show that genre-matching—using spoken language data to proxy for informal tweets—is more effective than simply increasing the volume of training data, resulting in a 21.6% improvement in F1 score over the baseline.
Introduction
Coreference resolution—the task of identifying all expressions that refer to the same entity in a text—is a fundamental component of Natural Language Processing (NLP). While performance on formal text has reached high levels, social media platforms like Twitter present unique challenges. Twitter's "perfect storm" of linguistic difficulties includes noisy language (abbreviations, hashtags, @mentions), shifting conversational contexts in multi-user threads, and exophoric references to external media.
Current state-of-the-art models are predominantly trained on the OntoNotes 5.0 dataset, which consists largely of formal news and magazines. This creates a significant domain gap when applied to the informal, rapid-fire discourse of Twitter. This research aims to bridge this gap through targeted domain adaptation.
Methods and Datasets
The researchers utilized the e2e-coref system, a neural resolver using ELMo embeddings, as the baseline. The primary methodology involved domain adaptation through strategic retraining across five distinct configurations (A through E). These configurations were designed to isolate the impact of data volume versus data genre.
The datasets used included:
- OntoNotes 5.0: The industry standard, containing 1.3 million tokens across genres like broadcast news, telephone conversations, and web blogs.
- Twitter Conversation (TW’): A specialized dataset of 185 Twitter threads (~48k tokens), annotated to align with OntoNotes standards.
As shown in Table 1, the configurations ranged from a baseline trained only on OntoNotes (Test A) to a "Twitter + Spoken" mix (Test D) and a "Twitter + Written" mix (Test E).
Key Insights
The experiments revealed several critical findings regarding how AI processes informal text:
- Genre Over Volume: Test D (Twitter + Spoken) outperformed Test C (Twitter + All OntoNotes), despite Test C having more data. This suggests that the formal news sections in OntoNotes introduce noise or bias that hinders performance on informal text.
- Mention Span Bias: Models trained on formal text tend to select longer, often incorrect, text spans as mentions. Adapted models learned to identify the shorter, more concise mentions typical of Twitter.
- Pronoun Resolution: Twitter is rich in 1st and 2nd person pronouns. The inclusion of Twitter-specific data significantly improved the resolution of "I," "me," and "you," which are vital for following conversational threads.
- Twitter-Specific Tokens: Retraining allowed the model to recognize hashtags and @usernames as valid coreferent entities rather than ignoring them.
Results
The performance gains from domain adaptation were substantial. The best-performing configuration, Test D (Twitter + Spoken), achieved an average F1 score of 66.8, representing a 21.6% improvement over the Test A baseline (45.18).
Notably, even training on the small Twitter-only dataset (Test B) yielded an F1 score of 60.8, significantly outperforming the massive OntoNotes-only baseline. This highlights the severe limitations of formal models when applied to social media. The adapted models also reduced missed entity errors by over 50%.
Conclusion
This study demonstrates that effective coreference resolution on Twitter requires more than just large datasets; it requires data that matches the linguistic register of the target domain. By leveraging spoken language data as a proxy for informal digital conversation, researchers can significantly improve model accuracy. Future work will further explore the nuances between "formal vs. informal" and "spoken vs. written" registers to refine these adaptation techniques.
Summary updated