TweetNERD - End to End Entity Linking Benchmark for Tweets

Shubhanshu Mishra, Aman Saini, Raheleh Makki, Sneha Mehta, Aria Haghighi, Ali MollahosseiniNeurIPS 2022

Introduction

TweetNERD is a massive new benchmark dataset designed to fix the "link rot" and temporal bias plaguing social media Natural Language Processing (NLP). With over 340,000 tweets annotated for Named Entity Recognition and Disambiguation (NERD) against Wikidata, it provides the largest and most temporally diverse playground for training models to understand the messy, short-form world of Twitter.

The Challenge of Twitter NERD

Named Entity Recognition and Disambiguation (NERD) is the two-step process of identifying a mention (e.g., "Apple") and linking it to a specific entry in a knowledge base (e.g., the tech company, not the fruit). While this is a largely solved problem for long-form news text, Twitter is notoriously difficult for NERD. Tweets are short, lack context, use non-standard grammar, and evolve rapidly.

Existing datasets for this task suffer from three major issues:

  1. Scale: Most are tiny, often containing only a few thousand tweets.
  2. Temporal Bias: They typically cover a very narrow window of time, meaning models trained on them fail when "trending" entities inevitably change.
  3. Link Rot: Because tweets are frequently deleted or made private, older benchmarks lose their validity over time as the underlying text becomes inaccessible via the Twitter API.
Table illustrating link rot: the percentage of tweets still 'Found' in older datasets.

Methodology

The authors developed a robust pipeline to create TweetNERD. Instead of relying on random sampling, they used Entropy-based sampling. By calculating the "disambiguation difficulty" of phrases based on Wikipedia page views, they were able to over-sample "hard" cases—entities that are ambiguous or less popular. This ensures the benchmark isn't just a collection of easy-to-guess celebrities.

For annotation, they utilized a trained in-house team rather than generic crowdsourcing. Annotators linked mentions to Wikidata, a massive, multilingual, and constantly updated knowledge base. They also implemented a strict "gold standard" system where at least two out of three annotators had to agree on a mention and its entity ID for it to be considered a ground-truth label.

The TweetNERD Dataset

The paper introduces three main components to the dataset, vastly outscaling previous efforts:

Comparison of TweetNERD's scale against previous benchmark datasets.
  • TweetNERD (Full): 340K+ unique tweets spanning 2010–2021, containing 356K mentions linked to 90K unique Wikidata entities.
  • TweetNERD-OOD (Out-of-Domain): A 25K subset specifically designed to test how well models generalize to difficult, highly ambiguous entities.
  • TweetNERD-Academic: A 30K subset where the authors took "classic" academic benchmarks (like WSDM2012 or NEEL) and re-annotated them using their modern, consistent guidelines. This allows for a fair comparison between old and new data.

Key Insights and Results

  • The Power of Re-annotation: One of the most valuable contributions is the "Academic" split. By re-labeling old datasets with consistent Wikidata IDs, the authors highlight how much "noise" existed in previous benchmarks due to inconsistent labeling rules.
  • The "Lookup" Baseline is Stubborn: In their evaluations, a simple heuristic—linking a mention to the most popular entity with that name (the "Lookup" method)—performed surprisingly well. When given the correct text spans, the Lookup baseline (F1 ~0.62) actually outperformed sophisticated models like GENRE and REL on the OOD split. This suggests that many current neural models still struggle to use the limited context of a tweet to outperform basic frequency statistics.
  • Temporal Sensitivity: The authors found that entity linking is highly sensitive to when a tweet was written. For example, "Academy Awards" in 2022 should link to the "94th Academy Awards," not just the general concept. TweetNERD is the first dataset large enough to actually test this temporal precision.
  • End-to-End Performance: Google’s Natural Language AI (NLAI) performed best overall for the full pipeline. However, for the NER task alone, top performers like Stanza and AllenNLP still only hovered around F1 scores of 0.50–0.55, showing that recognizing entities in tweets remains a very difficult challenge.

Limitations and Future Work

The primary limitation of TweetNERD is the non-static nature of the data. Because of Twitter's privacy policies, the authors can only distribute Tweet IDs, not the text itself. If users delete their tweets, those data points will eventually vanish from the benchmark, reintroducing the link rot problem over time.

Future work involves better utilizing the "social graph"—using information about who sent the tweet or what they were replying to—to provide the context that the 280-character limit lacks. The authors also suggest that the community needs to move toward "offset-agnostic" evaluation, as many errors in current systems come from slightly misaligned character counts rather than actually missing the entity entirely.

Summary updated

BibTeX
@inproceedings{mishra2022tweetnerd,
  author = {Shubhanshu Mishra and Aman Saini and Raheleh Makki and Sneha Mehta and Aria Haghighi and Ali Mollahosseini},
  title = {{TweetNERD - End to End Entity Linking Benchmark for Tweets}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure