IITRoorkee@SMM4H 2024 Cross-Platform Age Detection in Twitter and Reddit Using Transformer-Based Model

Thadavarthi Sankar, Dudekula Suraj, Mallamgari Reddy, Durga Toshniwal, Amit AgarwalACL 2024

TL;DR

This paper tackles the challenge of automatically identifying self-reported ages from social media posts on Twitter and Reddit. The researchers found that transformer-based models, particularly RoBERTa-Base, significantly outperform other approaches in this binary classification task, achieving an F1-score of 0.878. This work is crucial for enhancing the utility of social media data in fields like public health monitoring and targeted advertising.

What Problem Does This Paper Address?

Social media platforms like Twitter and Reddit are rich sources of information, but extracting specific demographic details, such as a user's age, can be challenging. This is particularly true for "self-reported exact age," where users explicitly state their age. The SMM4H (Social Media Mining for Health Applications) workshop aims to advance research in this area, and this paper focuses on Task 6: "Self-reported exact age classification with cross-platform evaluation in English." The core problem is to accurately distinguish between social media posts that contain explicit age information and those that do not, a task complicated by the similar language patterns observed in both categories, as illustrated by word clouds in Figure 1. This capability is vital for applications ranging from public health surveillance to personalized marketing.

Methods

The authors explored several transformer-based models, leveraging the Huggingface toolkit, for this binary classification task. The models investigated include:

  • RoBERTa-Base: This model was fine-tuned on the training dataset with specific hyperparameters (learning rate 2e-5, 10 epochs, batch size 64).
  • BERT-Base: Also fine-tuned using the Hugging Face library, with a learning rate of 2e-5 and a batch size of 16, trained for up to 4 epochs.
  • BiLSTM: This recurrent neural network architecture was used for its effectiveness in sequential data processing, employing a Sigmoid activation function and Adam optimizer, trained for up to 5 epochs.
  • Flan T5 Base: This versatile model was fine-tuned for text classification with a learning rate of 3e-4, batch size of 8, and trained for 2 epochs.

The core methodology involves fine-tuning these pre-trained language models on the provided datasets and evaluating their performance using standard metrics like F1-score, Precision, and Recall.

Datasets

The study utilized datasets provided by the SMM4H-2024 workshop, which include posts from both Twitter and Reddit. The overall dataset comprises approximately 13,200 tweets and 100,000 Reddit posts.

  • Training Data: 8,800 tweets from SMM4H22 and 100,000 unlabeled Reddit posts.
  • Validation Data: 2,200 tweets from SMM4H22 and 1,000 Reddit posts related to dry eye disease from SMM4H22.
  • Testing Data: 2,200 tweets from SMM4H22, 2,000 Reddit posts about dry eye disease from SMM4H22, and 12,482 Reddit posts about social anxiety.

The evaluation metrics used were standard F1-Score, Precision, and Recall.

Key Insights and Ideas

  • Subtle Language Differences: The word clouds for labeled (self-reported age) and unlabeled datasets look remarkably similar, highlighting the difficulty of distinguishing between them based on simple keyword analysis. This underscores the need for sophisticated NLP models.
  • Transformer Dominance: Transformer-based models, with their ability to understand contextual nuances, are well-suited for this task, outperforming traditional recurrent neural networks like BiLSTM.
  • RoBERTa's Superiority: RoBERTa-Base emerged as the top-performing model, demonstrating its robustness in accurately identifying age-related information even in subtle contexts.
  • Cross-Platform Applicability: The task's focus on cross-platform evaluation (Twitter and Reddit) is crucial for building generalizable models that can be applied across diverse social media landscapes.

Key Results

The RoBERTa-Base model achieved the highest performance on the validation dataset, with an F1-Score of 0.88, a Precision of 0.899, and a Recall of 0.858. When submitted to the test dataset, the RoBERTa-Base model achieved an F1-score of 0.878, a Precision of 0.899, and a Recall of 0.858. Table 3 provides an example of how RoBERTa correctly identified three out of four sample tweets, showcasing its strong performance, though it did misclassify one example.

Limitations and Future Work

The authors acknowledge several limitations:

  • Occasional Misclassification: Even the best model (RoBERTa) occasionally misclassified tweets, particularly those with subtle context differences or implicit age information.
  • Token Limits: Transformer-based models have inherent token limits, which can pose challenges for processing very long posts.

Future work suggested includes:

  • Developing Effective Chunking Methods: To handle longer texts and improve classification accuracy.
  • Addressing Implicit Age Information: Further research is needed to better capture age information that is not explicitly stated.
  • Handling Diverse Linguistic Constructs: Improving robustness to variations in language use and slang.
  • Model Robustness: Continued focus on enhancing model robustness for real-world applications.

Summary updated

BibTeX
@inproceedings{sankar2024iitroorkee,
  author = {Thadavarthi Sankar and Dudekula Suraj and Mallamgari Reddy and Durga Toshniwal and Amit Agarwal},
  title = {{IITRoorkee@SMM4H 2024 Cross-Platform Age Detection in Twitter and Reddit Using Transformer-Based Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure