TL;DR
This paper tackles the challenge of automatically detecting abusive language in Russian, a task complicated by the language's rich morphology and lexical diversity. The authors introduce a new dataset and explore various machine learning approaches, achieving state-of-the-art performance comparable to English models, even with a simple SVM.
What Problem Does This Paper Address?
The internet is rife with abusive language, creating toxic online environments and posing moderation challenges for social media platforms. While significant research exists for English, abusive language detection in other languages, particularly Russian, remains less explored. This is due to a scarcity of labeled data and the inherent linguistic complexities of Russian, such as its flexible word order, rich morphology, and extensive use of slang and diminutives. This paper aims to bridge this gap by developing and evaluating methods for recognizing abusive language specifically in Russian.
Methods
The authors employ a multi-stage preprocessing pipeline for text data, including:
- Dataset Balancing: Undersampling the majority "non-hate" class to match the "hate" class proportion.
- Text Cleaning: Removing URLs and adjusting platform-specific tokens (like @mentions and #hashtags).
- Orthographic Normalization: Standardizing Russian characters (e.g., 'ë' to 'e').
- Tokenization: Splitting text into words and punctuation using NLTK.
- Lemmatization: Reducing words to their base form using
pymorphy2. - Stop Word Removal: Eliminating common words that don't carry significant meaning.
- TF-IDF Vectorization: Converting text into numerical feature vectors.
- Train-Test Split: Dividing data into training and testing sets (80/20).
Beyond standard text features, they extract:
- Sentiment Analysis: Leveraging pre-trained sentiment models, assuming abusive language often correlates with negative sentiment.
- Upper-casing Count: Measuring the frequency of fully capitalized words as an indicator of heightened tone.
- Offensive Word Count: Counting occurrences of words from a custom-built abusive language lexicon.
They evaluate several models, including a baseline Linear Support Vector Classifier (LinearSVC) and deep learning models like RuBERT and mBERT.
Datasets
The paper introduces a new, manually annotated dataset for Russian abusive language detection. This dataset is a combination of:
- Russian South Park (RSP): Subtitles from the TV show "South Park," manually annotated for abusive language. This source was chosen for its high density of profanity and hate speech.
- Russian Troll Tweets: A large repository of tweets, filtered for Cyrillic text, from which a subset was manually labeled.
- RuTweetCorp: Another Russian tweet dataset that was also annotated.
- Russian Language Toxic Comments (RTC): A publicly available dataset containing over 14,000 labeled samples of hate speech.
The combined RSP+RTC dataset forms the primary benchmark for their experiments.
Key Insights and Ideas
- Linguistic Diversity is a Major Hurdle: The paper strongly emphasizes that Russian's grammatical and lexical structure presents unique challenges compared to English, necessitating careful preprocessing and feature engineering.
- Data Quality and Quantity are Crucial: The scarcity of labeled Russian data is a significant bottleneck. The creation and annotation of the RSP dataset, combined with existing resources, were vital for achieving meaningful results.
- Preprocessing Matters Immensely: The authors demonstrate that careful data cleaning, normalization, and feature selection (like stop word removal) significantly impact model performance, even for simple models like SVM.
- Simple Models Can Be Powerful: A LinearSVC, when combined with effective preprocessing and feature extraction, achieves surprisingly strong results, suggesting that complex deep learning models aren't always necessary for high performance in this domain.
- Domain-Specific Models Outperform Multilingual Ones: RuBERT, a BERT model specifically trained on Russian data, outperforms mBERT (a multilingual BERT) on the Russian abusive language task, highlighting the benefits of language-specific pre-training.
Key Results
- The baseline LinearSVC model achieved an F1-score of up to 0.75 on the RSP dataset and 0.88 on the combined RSP+RTC dataset.
- Removing stop words improved the F1-score for the LinearSVC model to 0.88 on the RSP+RTC dataset.
- RuBERT achieved an F1-score of 0.85 on the combined dataset, outperforming mBERT which scored 0.76.
- The paper suggests that with proper preprocessing and feature selection, state-of-the-art performance comparable to English models can be achieved for Russian abusive language detection.
Limitations and Future Work
- Model Struggles with Complex Cases: RuBERT, despite its strong performance, still struggles with recognizing abusive language in longer texts and those containing misspellings or many named entities.
- mBERT's Generalization Issue: The multilingual nature of mBERT, while beneficial for broad language coverage, seems to dilute its effectiveness for the specific nuances of Russian abusive language compared to a dedicated Russian model.
- Future Directions: The authors suggest that further research could focus on improving the handling of complex linguistic phenomena in Russian, such as slang and misspellings, and exploring more sophisticated feature extraction techniques. The success with SVM also opens avenues for exploring simpler, more interpretable models.
Summary updated