Abstract
User-generated text on social media often suffers from undesired characteristics, including hatespeech, abusive language, and insults targeted at specific groups. Such text frequently deviates from traditional linguistic norms through the use of explicit abuse, obfuscated words, typographical errors, or implicit negative stereotypes. Processing this text poses significant robustness challenges for standard Natural Language Processing (NLP) techniques. This paper analyzes how character, subword, and Byte Pair Encoding (BPE) models can address these challenges. We evaluate the effectiveness of various word decomposition techniques, experiment with fine-tuning large pretrained language models, and demonstrate their robustness to domain shift across Wikipedia attack, toxicity, and Twitter hatespeech datasets.
Introduction: The Challenge of Noisy Text
Standard NLP models typically operate on a fixed vocabulary of whole words. While effective for formal text like news articles, this approach fails on social media, where users often intentionally obfuscate language to evade automated filters. Common tactics include typos (e.g., "harrassment"), character substitutions (e.g., "w0m3n"), or creative slurs.
When a word-based model encounters an unseen string—an "Out-of-Vocabulary" (OOV) token—it generally treats it as a generic unknown, losing all semantic meaning. This research investigates "Neural Word Decomposition," a suite of techniques that look inside the word to identify meaningful sub-units, ensuring that obfuscated terms like "w0m3n" are correctly associated with their root concepts rather than being discarded as gibberish.
Methodology and Architectures
The study compares several architectures to determine which decomposition strategy best handles noisy user-generated content:
- Word-based (Baseline): Utilizing fastText, which represents documents by averaging word embeddings.
- Subword-based: An extension of fastText that considers character n-grams within words (e.g., "apple" decomposed into "ap", "app", "ppl").
- Joint Word and Character Embedding (TextCNN): A model using a Convolutional Neural Network (CNN) to extract character-level features, which are then concatenated with standard word embeddings.
- End-to-End Character Model (VDCNN): A "Very Deep" CNN that operates entirely at the character level, bypassing word boundaries.
- Pretrained Language Models (BERT): Fine-tuning BERT-base, which utilizes WordPiece tokenization to break words into frequent sub-units.
A primary contribution of this work is testing the BERT WordPiece tokenizer as a standalone preprocessing step for simpler models. This allows researchers to determine if the robust word-splitting logic of BERT can "supercharge" computationally efficient models like fastText without requiring the full overhead of a transformer architecture.
Key Results and Insights
The models were evaluated on three benchmarks: Twitter Hatespeech (16k tweets), Wikipedia Personal Attacks (W-ATT, 115k comments), and Wikipedia Toxicity (W-TOX, 159k comments).
The "Tokenizer Transfer" Effect
One of the most significant findings was that preprocessing text with BERT’s WordPiece model before feeding it into a simple fastText model led to substantial performance gains. On the Wikipedia Toxicity set, the baseline fastText model scored 86.8; adding BERT-style tokenization boosted this to 88.6, rivaling much more complex architectures. This suggests that BERT's method of seeing subwords is inherently robust to the noise found in abusive language.
Performance Hierarchy
- BERT is King: Fine-tuned BERT achieved the highest Macro F1 scores on the Wikipedia datasets (89.5 for W-ATT and 90.6 for W-TOX).
- Word Boundaries Matter: Purely character-based models (VDCNN) performed worse than models that maintained some concept of word boundaries. This indicates that while sub-word information is vital, the "word" remains a fundamental unit of meaning for detecting intent.
- BPE Robustness: BPE models trained on clean data (Wikipedia) transferred remarkably well to noisy social media text. By breaking unknown obfuscated words into known sub-units, the models successfully identified terms like "mother-fucker" (obfuscated with a hyphen) and "idiotic" (broken into idiot + ic).
Conclusion and Future Work
While neural word decomposition significantly improves the detection of obfuscated abuse, challenges remain regarding implicit abuse—language that is offensive due to context or stereotypes rather than specific keywords. Furthermore, the high variance in results for certain categories (like racism in the Twitter dataset) highlights the ongoing issue of data bias. Future research may focus on pre-training models directly on massive social media corpora to better capture the unique linguistic evolution of online communities.
Summary updated