Broad Twitter Corpus: A Diverse Named Entity Recognition Resource

Leon Derczynski, Kalina Bontcheva, Ian RobertsCOLING 2016 · OpenReview

Abstract

One of the main obstacles hampering method development and comparative evaluation of named entity recognition (NER) in social media is the lack of a sizeable, diverse, high-quality annotated corpus. This paper introduces the Broad Twitter Corpus (BTC), which is significantly larger than previous resources and sampled across different regions, temporal periods, and types of Twitter users. The gold-standard annotations are produced via a hybrid expert-crowd model, ensuring both high recall and high quality.

Introduction: The Challenge of Social Media NER

Named Entity Recognition (NER)—the identification of people, locations, and organizations—is a mature task for stable media like news articles. However, performance often degrades when applied to social media. The authors identify three primary hurdles:

  1. Data Scarcity: Existing Twitter NER datasets are small. For example, the Ritter corpus is only about 15% the size of the standard CoNLL’2003 newswire benchmark.
  2. Lack of Diversity: Datasets are often "snapshots" from a single time period or region, failing to capture the breadth of global discourse.
  3. Entity Drift: Language on social media evolves rapidly. A model trained on 2010 data may struggle with 2024 entities. This temporal, geographic, and social "drift" necessitates more diverse training data.

Methodology: Stratified Sampling and Hybrid Annotation

To address these issues, the researchers employed a stratified sampling approach for the BTC, collecting tweets across:

  • Time: A six-year span (2009–2014).
  • Space: Six English-speaking regions (UK, US, Canada, Ireland, NZ, and Australia).
  • Social Strata: Content from influencers ("Twitterati"), mainstream news, and random "non-directed" users.

Annotation followed a hybrid Expert + Crowd model. The authors noted an "Expert Blindspot": while experts follow strict rules, a diverse crowd is often better at recognizing niche pop-culture entities. By using a "max-recall" adjudication rule—where crowd-identified spans were reviewed by experts—they captured the "long tail" of entities without sacrificing precision.

The Broad Twitter Corpus (BTC)

The resulting BTC consists of 9,551 documents and 165,739 tokens, featuring over 12,000 entity mentions across Person (PER), Location (LOC), and Organization (ORG) categories.

Comparison of the Broad Twitter Corpus with existing social media NER datasets.

As shown in the table above, the BTC is nearly four times larger than the Ritter (2011) corpus and provides a more comprehensive entity schema and annotation process than its predecessors.

Key Insights: Entity Distribution and Local Context

A critical finding of the study is the difference in entity distribution between newswire and social media.

Surface form frequency distributions for Newswire vs. Social Media.

In newswire, a few "head" entities (e.g., "U.S." or "Clinton") dominate. In contrast, social media exhibits a much flatter distribution with a longer tail of unique entities. This means models must be prepared for a significantly wider variety of unique names than those found in traditional media.

Furthermore, the study highlighted that Spatial Context Matters: annotators are significantly more accurate when labeling tweets from their own geographic region. In some regions, over 80% of the entities found were "novel" (unique to that region), emphasizing the need for geographic stratification in training data.

Results and Future Directions

The BTC achieves a high inter-annotator agreement (IAA) of 0.94 after expert adjudication. Its temporal breadth—spanning six years compared to the single-day snapshot of the Ritter dataset—provides a clearer picture of entity evolution.

While currently limited to English and the PLO schema (dropping a "Product" category due to low agreement), the BTC serves as a robust benchmark for testing model generalization across time and space. Future work may involve using this corpus to quantify "anachronistic" model decay as training data ages.

Summary updated

Abstract

One of the main obstacles, hampering method development and comparative evaluation of named entity recognition in social media, is the lack of a sizeable, diverse, high quality annotated corpus, analogous to the CoNLL’2003 news dataset. For instance, the biggest Ritter tweet corpus is only 45,000 tokens – a mere 15% the size of CoNLL’2003. Another major shortcoming is the lack of temporal, geographic, and author diversity. This paper introduces the Broad Twitter Corpus (BTC), which is not only significantly bigger, but sampled across different regions, temporal periods, and types of Twitter users. The gold-standard named entity annotations are made by a combination of NLP experts and crowd workers, which enables us to harness crowd recall while maintaining high quality. We also measure the entity drift observed in our dataset (i.e. how entity representation varies over time), and compare to newswire. The corpus is released openly, including source text and intermediate annotations.
BibTeX
@inproceedings{derczynski2016broad,
  author = {Leon Derczynski and Kalina Bontcheva and Ian Roberts},
  title = {{Broad Twitter Corpus: A Diverse Named Entity Recognition Resource}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=F66fzwXBTBX},
}
Figure