A Comparative Study on Word Embeddings and Social NLP Tasks

Fatma Elsafoury, Steven R. Wilson, Naeem RamzanNAACL 2022

TL;DR

This paper investigates whether word embeddings trained on social media data (like Twitter or Urban Dictionary) are better suited for social NLP tasks like cyberbullying detection and measuring social bias, compared to traditional embeddings trained on formal text. The findings suggest social media embeddings offer improved performance in detecting offensive language and cyberbullying, though their social bias remains a complex and inconclusive area.

What Problem Does This Paper Address?

Natural Language Processing (NLP) tasks often rely on word embeddings, which are numerical representations of words that capture semantic relationships. Traditionally, these embeddings are trained on large, formal text corpora like news articles or Wikipedia (termed "informational-based"). However, the rise of social media platforms has led to the development of embeddings trained on more informal, diverse, and often unfiltered text (termed "social-media-based"). These platforms are known to contain linguistic nuances, slang, and offensive language not found in formal text.

Despite the potential of social-media-based embeddings to better understand the language used in social contexts, they have been under-explored for critical social NLP tasks like cyberbullying detection and the analysis of social biases. This paper aims to bridge this gap by systematically comparing the performance of social-media-based and informational-based word embeddings on these two tasks.

Methods

The researchers compared several pre-trained static word embeddings, including:

  • Informational-based: Word2Vec, GloVe-Wikipedia
  • Social-media-based: GloVe-Twitter, Urban Dictionary (UD), Chan

They evaluated these embeddings in two main ways:

  1. Offense Categorization: They used the Hurtlex lexicon, which categorizes offensive words into 11 groups relevant to cyberbullying. They trained a K-Nearest Neighbors (KNN) model to predict Hurtlex categories based on word embeddings and measured performance using F1 scores.
  2. Cyberbullying Detection: They initialized the embedding layer of a Bi-directional LSTM (Bi-LSTM) deep learning model with the different word embeddings and trained it to detect cyberbullying on various social media datasets. Performance was measured using F1 scores.
  3. Social Bias Measurement: They employed the WEFE framework and standard bias metrics (WEAT, RNSB, RND, ECT) to quantify gender and racial biases within the word embeddings.

Datasets

The paper utilized the following datasets:

  • For Offense Categorization: The Hurtlex lexicon (Zhang et al., 2020), which contains 8228 offensive words organized into 17 groups. For their experiments, they focused on 11 groups relevant to cyberbullying.
  • For Cyberbullying Detection: Five publicly available cyberbullying datasets were used:
  • Twitter-Racism (Waseem and Hovy, 2016)
    • Twitter-Sexism (Waseem and Hovy, 2016)
    • HateEval (Basile et al., 2019) - English tweets only
    • Kaggle (Kaggle, 2012)
    • Jigsaw (Jigsaw, 2018)
  • For Social Bias Measurement: Standard word lists and attribute lists were used as defined by existing bias measurement frameworks (WEAT, RND, RNSB, ECT).

Key Insights and Ideas

  • Social Media Embeddings Excel at Offensive Language: Social-media-based embeddings, particularly Urban Dictionary, showed a superior ability to group words belonging to similar offensive categories (like PS, OM, PR, ASF, ASM, DDP, DDF). This suggests they better capture the semantic nuances of insults and profanity.
  • Cyberbullying Detection Boost: Social-media-based embeddings generally outperformed informational-based ones in detecting cyberbullying across multiple datasets. This indicates their ability to capture the specific linguistic patterns of online harassment.
  • No Universal "Best" for Specific Cyberbullying Types: While social media embeddings were better overall for cyberbullying detection, the study found no strong evidence that specific embeddings were consistently better at detecting particular categories of cyberbullying within the datasets. This might be due to the low prevalence of certain categories or the complexity of the task.
  • Social Bias is Complex and Inconclusive: While the hypothesis that social-media-based embeddings are more biased was partially supported by some metrics (RND, RNSB, ECT), the results were not consistent across all metrics and types of bias. This highlights the challenges in definitively measuring and comparing social bias in word embeddings.

Key Results

  • Offense Categorization: Urban Dictionary embeddings performed best for most Hurtlex categories related to offensive language.
  • Cyberbullying Detection:
    • Glove-Twitter achieved the highest average F1 score (0.620) on HateEval.
  • Glove-Twitter also achieved the highest average F1 score (0.519) on Jigsaw-toxicity.
  • Chan embeddings achieved the highest average F1 score (0.727) on Kaggle.
  • Urban Dictionary embeddings achieved the highest average F1 score (0.663) on Twitter-racism.
  • Glove-Wikipedia (informational-based) performed best on Twitter-sexism (0.699).
  • Overall, social-media-based embeddings outperformed informational-based ones on four out of five datasets, with significant differences in three cases.
  • Social Bias: Most bias metrics (RND, RNSB, ECT) indicated Chan and Urban Dictionary as highly biased for gender and racial bias. However, Glove-WK (informational-based) was also ranked highly by some metrics, leading to inconclusive findings about social-media-based embeddings being universally more biased.

Limitations and Future Work

  • Category Prevalence: The study noted that the low percentage of certain Hurtlex categories in the cyberbullying datasets might have limited the ability to reliably assess the performance of embeddings on specific offense types.
  • Bias Measurement Inconsistency: The lack of agreement between different bias metrics makes it difficult to definitively rank embeddings by their social bias.
  • Future Work: The authors suggest further investigation into the relationship between word embedding bias and their performance on cyberbullying detection. They also propose that profanity should be considered as a crucial feature alongside others for more robust cyberbullying detection models and for revealing social bias. More research is needed to understand how social bias manifests in different forms and how it impacts NLP tasks.

Summary updated

BibTeX
@inproceedings{elsafoury2022comparative,
  author = {Fatma Elsafoury and Steven R. Wilson and Naeem Ramzan},
  title = {{A Comparative Study on Word Embeddings and Social NLP Tasks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure