TL;DR
This paper introduces the first corpus and study of pronominal anaphora resolution specifically within Twitter conversations. By analyzing a novel dataset of 185 Twitter threads, the authors identify unique linguistic phenomena and evaluate an off-the-shelf coreference resolution system, revealing significant challenges for current models in this dynamic social media context.
What Problem Does This Paper Address?
Resolving anaphora – figuring out what pronouns like "he," "she," or "it" refer to – is a fundamental task in natural language processing. While much research has focused on this in formal text or spoken dialogue, the unique characteristics of Twitter conversations present a new frontier. Twitter's conversational structure, often characterized by short, rapid exchanges, embedded media, and informal language, makes anaphora resolution particularly challenging. Existing systems trained on more traditional text may struggle with phenomena like pronouns referring directly to speakers, exophoric references to external content (images, links), and the influence of conversation structure on how references are understood. This paper aims to bridge this gap by exploring these challenges and providing a foundation for future research in this area.
Methods
The authors constructed a novel corpus of 185 Twitter conversation threads, carefully annotating pronominal anaphora. They focused on 3rd person singular pronouns (he, him, his, himself, she, her, herself, it, its, itself) as they are most relevant for anaphora resolution. The annotation process involved manual identification of pronouns and their antecedents, considering the entire reference chain. To prepare the data for NLP systems, they experimented with and combined two tokenizers, Stanford's PTBTokenizer and Twokenizer, to handle Twitter's unique text formatting. Finally, they evaluated the performance of the Stanford statistical coreference resolution system on their annotated data, performing a qualitative error analysis to understand the system's shortcomings.
Datasets
The primary contribution is a newly created corpus of 185 Twitter conversation threads. This corpus was collected using the twarc tool and filtered to include threads with at least three tweets and at least one instance of a 3rd person singular pronoun. The corpus is annotated for coreference chains, including pronouns, proper names, and full NPs. The authors also used the CONLL 2012 scorer for evaluating the Stanford system's output against their annotated data.
Key Insights and Ideas
- Twitter conversations are a distinct genre: The informal, multi-modal, and structurally complex nature of Twitter threads introduces unique anaphoric phenomena not typically found in standard text.
- Conversation structure matters: The way tweets are threaded (reply-to relationships) significantly influences how pronouns are understood, a factor that standard resolvers may not adequately capture.
- Beyond text: Exophoric references are common: Pronouns can refer to entities in embedded images, videos, or linked URLs, requiring systems to go beyond textual analysis.
- Tokenization is crucial: Twitter's informal language, including misspellings, abbreviations, and non-standard punctuation, necessitates robust preprocessing. The authors found that combining different tokenizers was beneficial.
- Off-the-shelf systems struggle: Even a sophisticated system like Stanford's coreference resolver shows significant error rates when applied to this data, highlighting the need for Twitter-specific adaptations.
Key Results
The evaluation of the Stanford coreference resolution system on the annotated Twitter data yielded mixed results. While the system achieved an F1 score of 50.01% (using CEAFE metric) when provided with speaker and turn information, the error analysis revealed substantial challenges. The most common error types included:
- Wrong or missing antecedents in chains (39% of errors): The system often assigned incorrect antecedents or failed to find any.
- Missing matches due to lack of world knowledge (23% of errors): The system struggled to connect entities that require external knowledge (e.g., knowing "Hillary Clinton" is "The Secretary of State").
- Missing matches due to hashtags and @-mentions (7% of errors): The system failed to recognize that hashtags and usernames could refer to the same entity as a pronoun.
- Incorrect mention spans (36% of errors): The system had difficulty correctly identifying the boundaries of mentions, especially when usernames or hashtags were involved.
Limitations and Future Work
The authors acknowledge several limitations and outline future research directions:
- Focus on 3rd person pronouns: The study did not cover 1st and 2nd person pronouns, or plural pronouns, which are left for future work.
- Non-aligning replies: The paper notes the potential issue of users replying to the wrong tweet in a thread, which is hard to detect automatically and was not deeply investigated.
- Limited scope of evaluation: The evaluation focused on a subset of coreference chains (those with 3rd person singular pronouns) to make the Stanford system's output comparable to their annotations.
- Future work: The authors plan to experiment with different preprocessing variants to measure their effect on resolver performance. They also aim to identify fundamental problems that remain for resolvers trained on standard text when applied to Twitter conversations and explore ways to tackle them.
Summary updated