Abstract
Toxic language detection in social media typically focuses on post-level classification, often overlooking the nuanced lexical shifts of specific words. This research explores pejorative language—terms used to disparage or belittle—by framing its detection as a Word Sense Disambiguation (WSD) task. We leverage contextual embeddings from transformer models (e.g., BERT, RoBERTa) to distinguish between neutral and pejorative uses of words. Our results, spanning English, Spanish, Italian, and Romanian, demonstrate that contextual models can effectively isolate pejorative senses, achieving F1-scores up to 0.864 on newly curated English datasets.
Introduction
In Natural Language Processing (NLP), "toxic language" is a broad umbrella term encompassing hate speech, aggression, and offensiveness. However, pejorativity represents a distinct linguistic category that is highly context-dependent. For instance, the word "trash" can function as a literal noun (neutral) or a figurative slur (pejorative). Existing post-level models often fail to capture these lexical nuances, leading to false positives in moderation systems. This study seeks to bridge this gap by focusing on word-level disambiguation.
Methodology
The research follows a three-phase approach:
- Lexicon Construction: Using online dictionaries like Wiktionary and dexonline.ro, we identified terms explicitly marked as "derogatory" or "pejorative" across four languages.
- WSD Framework: Pejorativity detection is treated as a binary classification task. Given a target word and its context (e.g., a tweet), the model determines if the usage is pejorative.
- Contextual Embeddings: We utilized transformer-based models—including BERT, RoBERTa, BERTweet, and Multilingual BERT—to extract hidden states for target words. These embeddings were then processed through various classifiers (k-NN, SVM, and MLP).
- Unsupervised Clustering: For languages with limited labeled data, such as Spanish, we applied K-Means clustering to embeddings to evaluate the models' inherent ability to separate word senses.
Datasets
We introduce several new resources for the community:
- Multilingual Lexicon: A collection of pejorative terms for English (2,903), Spanish (881), Italian (149), and Romanian (770).
- PEJOR1 & PEJOR2: Two English datasets derived from existing hate speech corpora. PEJOR1 preserves original label distributions, while PEJOR2 is balanced for pejorative vs. non-pejorative senses.
- Spanish Dataset: An expert-annotated subset focusing on 12 pejorative words.
Analysis and Insights
A core finding is that pejorative language is primarily a lexical phenomenon distinct from general offensiveness. The "power of context" is most evident when visualizing how models like BERT cluster different meanings.
As shown in the visualization for the word "cracker," contextual embeddings naturally separate the literal sense (food) from the pejorative sense (slur). This separation is crucial for reducing false positives in automated moderation. Furthermore, the study notes the challenge of semantic amelioration, where words like "queer" lose their pejorative weight over time, requiring models to be updated continuously.
Key Results
The experimental results confirm the efficacy of contextual models in identifying pejorative senses.
In English, BERT base combined with a Multilayer Perceptron (MLP) achieved the highest performance on the PEJOR2 dataset with an F1-score of 0.864. BERTweet also performed strongly, reaching an F1 of 0.802. In Spanish, even without labeled training data, the BETO model achieved 68.9% accuracy via unsupervised clustering, highlighting the robust cross-lingual potential of these embeddings.
Conclusion and Future Work
This work demonstrates that word-level pejorativity detection is a viable and necessary component of toxic language analysis. While English results are robust, future work must address data scarcity in other languages and the complexities of cross-lingual translation of slurs. Moving forward, we aim to transition from single-word classification to sequence labeling and integrate these detectors into broader hate speech detection systems to enhance both accuracy and explainability.
Summary updated