SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research

Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, Leonardo Neves, Kiamehr Rezaee, Luis Espinosa Anke, Jiaxin Pei, José Camacho-Collados

Abstract

Despite the rapid evolution of natural language processing (NLP), the maturity of models and benchmarks for social media remains significantly behind general-purpose alternatives. The fragmented landscape of social media NLP makes it difficult to identify state-of-the-art performance across diverse tasks. This paper introduces SuperTweetEval, a unified benchmark designed to evaluate NLP models on the idiosyncratic and heterogeneous nature of social media text. Comprising 12 tasks—including emoji prediction, sentiment analysis, and question answering—SuperTweetEval provides a standardized framework for assessment. Our benchmarking of various architectures, from masked language models to large-scale generative models, reveals that social media remains a formidable challenge, with domain-specific pre-training still offering a distinct advantage over general-purpose large language models (LLMs).

1. Introduction

Social media platforms represent a unique linguistic frontier characterized by non-standard grammar, evolving slang, and heavy reliance on visual cues like emojis. While benchmarks such as GLUE and SuperGLUE have successfully driven progress in general language understanding, social media NLP has historically suffered from fragmentation. Evaluation has often been limited to isolated tasks, primarily sentiment classification, failing to capture the complexity of real-world social media interactions.

SuperTweetEval addresses this gap by unifying 12 diverse tasks into a single, heterogeneous benchmark. By providing a standardized test bed, the researchers aim to facilitate clearer comparisons between models and highlight the specific areas where current NLP technologies struggle in the context of social media.

2. The SuperTweetEval Benchmark

The benchmark consists of 255,170 tweets across 12 datasets. These datasets are categorized into three groups:

2.1 Existing Datasets

The benchmark incorporates established datasets for Intimacy Analysis, Meaning-Shift (TempoWiC), Sentiment Analysis, Emotion Detection, Topic Classification, Question Answering (QA), and Named Entity Recognition (NER).

2.2 Adapted Datasets

Several datasets were reframed to fit the unified benchmark:

  • TweetNERD: Adapted for entity disambiguation.
  • TweetHate: A robust dataset for hate speech detection.
  • TweetQG: Focused on Question Generation.

2.3 New Datasets

Two new datasets were constructed from scratch to address specific social media nuances:

  • TweetEmoji100: This dataset expands emoji prediction from the traditional 20-emoji set to 100, reflecting the true diversity of emoji usage.
  • TweetSim: A dataset for tweet similarity, requiring models to score the relationship between two tweets on a scale of 0-5.

3. Methodology

To ensure consistency, all data was unified into a single JSON format with standardized preprocessing, including the masking of URLs and mentions to preserve privacy. The researchers evaluated three primary modeling approaches:

  1. Fine-tuning Masked Language Models (MLMs): Models like RoBERTa and TimeLM (pre-trained on Twitter data across different temporal snapshots) were fine-tuned on the benchmark tasks.
  2. Fine-tuning Generative Models: Encoder-decoder architectures (e.g., FlanT5) and decoder-only models (e.g., OPT) were evaluated.
  3. In-Context Learning (ICL): The zero-shot and few-shot capabilities of large-scale models, specifically GPT-3.5 (ChatGPT), were tested to assess the performance of general-purpose LLMs without task-specific fine-tuning.

4. Key Results and Insights

The benchmarking process yielded several critical insights into the current state of social media NLP:

  • The Dominance of Domain-Specific Pre-training: TimeLM-Large emerged as the overall winner, achieving the highest average score (0.63) across all tasks. This suggests that pre-training on domain-specific data (Twitter) provides a significant architectural advantage over general-purpose models of similar size.
  • The Generative Gap: While generative models like FlanT5 performed exceptionally well in Question Answering, they lagged behind traditional MLMs in classification and regression tasks.
  • The Challenge of Large Label Spaces: Tasks with high-cardinality outputs, such as TweetEmoji100, proved particularly difficult. Models struggled to capture the subtle contextual nuances required to select the correct emoji from 100 options.
  • LLM Performance Disparity: While GPT-3.5 performed competently in Hate Speech detection and QA, its performance dropped by approximately 50% on regression tasks (e.g., Intimacy and Similarity) compared to smaller, fine-tuned models.
  • Temporal Sensitivity: The benchmark's temporal tasks highlighted that social media language is a "moving target," where models must account for rapid shifts in usage over time.

5. Conclusion and Future Work

SuperTweetEval demonstrates that despite the impressive capabilities of modern LLMs, social media remains a challenging domain that benefits from specialized attention. The benchmark identifies emoji prediction and question generation as particularly difficult "frontier" tasks.

Future iterations of the benchmark aim to address current limitations, including the expansion to multilingual datasets to reflect the global nature of social media. Additionally, the researchers plan to explore the inclusion of multiple datasets per task to better capture variance in complex areas like hate speech detection. SuperTweetEval stands as a vital tool for the community, providing a rigorous and unified standard for the next generation of social media NLP research.

Summary updated

Abstract

Despite its relevance, the maturity of NLP for social media pales in comparison with general-purpose models, metrics and benchmarks. This fragmented landscape makes it hard for the community to know, for instance, given a task, which is the best performing model and how it compares with others. To alleviate this issue, we introduce a unified benchmark for NLP evaluation in social media, SuperTweetEval, which includes a heterogeneous set of tasks and datasets combined, adapted and constructed from scratch. We benchmarked the performance of a wide range of models on SuperTweetEval and our results suggest that, despite the recent advances in language modelling, social media remains challenging.
BibTeX
@inproceedings{antypas2023supertweeteval,
  author = {Dimosthenis Antypas and Asahi Ushio and Francesco Barbieri and Leonardo Neves and Kiamehr Rezaee and Luis Espinosa Anke and Jiaxin Pei and José Camacho-Collados},
  title = {{SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=USAa0g47ja},
}
Figure