TL;DR
This paper tackles the challenging task of automatically detecting cyberaggression in social media text, a crucial step for mitigating its harmful effects. The authors propose a deep learning approach combining multiple feature sets and models, demonstrating that while overt aggression is detectable, covert forms remain a significant hurdle, highlighting the need for richer contextual information.
What Problem Does This Paper Address?
Cyberaggression and cyberbullying are pervasive issues on social media, with harmful consequences for individuals. The sheer volume of online content makes manual moderation impossible, necessitating automated detection systems. However, detecting aggression is complex due to several factors:
- Evolving Language: Aggressive language can be disguised, using sarcasm or irony, making it hard to detect with traditional methods.
- Context Dependency: A seemingly innocuous post can become aggressive when viewed within a larger conversation or thread.
- Informal Language: Social media posts often lack grammatical correctness and proper syntax, posing challenges for Natural Language Processing (NLP) tools.
- Subtlety: Aggression can manifest in subtle ways, targeting personal attributes or using veiled insults, making it difficult to identify based solely on keywords.
This paper aims to develop an intelligent system capable of automatically detecting harmful content across various social media platforms, specifically addressing the challenge of identifying both overt and covert forms of aggression.
Methods
The proposed approach, developed by the "Isistanitos" team, integrates multiple feature extraction techniques with two well-established classification models: a neural network and a Support Vector Machine (SVM).
Feature Extraction: * GloVe Features: Word embeddings generated using GloVe (Global Vectors for Word Representation) are used to represent words as dense vectors, capturing semantic relationships. * Sentiment Features: Sentiment scores (positivity, negativity, objectivity) derived from SentiWordNet are used to represent the emotional tone of words. * Composed Features: This set combines TF-IDF (Term Frequency-Inverse Document Frequency) for word stems, sentiment analysis, and punctuation-related features. * N-gram TF-IDF Features: This approach extends TF-IDF to include character n-grams (3-grams, 4-grams, and 5-grams) alongside word stems, aiming to capture misspellings and variations in language.
Predictive Model: * Neural Network: This model utilizes the GloVe, Sentiment, and Composed features. It employs Gaussian Noise layers for data augmentation, followed by LSTM (Long Short-Term Memory) layers for sequence processing (for GloVe and Sentiment features) and dense layers for Composed features. A final dense layer with softmax activation provides class probabilities. * Support Vector Machine (SVM): This model is trained on the N-gram TF-IDF features, leveraging an RBF kernel.
The final prediction is an average of the class probabilities from both the neural network and the SVM.
Datasets
- Kumar et al. (2018b) Dataset: This dataset, originally comprising Hindi and English posts from Facebook and Twitter, was used for training and validation. For this study, only the English posts were utilized. The dataset was manually tagged into three aggression levels: Overt Aggressive (OAG), Covert Aggressive (CAG), and Non-Aggressive (NAG).
- Reynolds et al. (2011) Dataset: This dataset, consisting of approximately 3,000 questions and answers from FormSpring.me, was used for evaluation. Posts were labeled as strongly aggressive, weakly aggressive, and non-aggressive. This dataset was randomly split into 70% for training and 30% for testing.
Key Insights and Ideas
- Covert Aggression is the Achilles' Heel: The paper strongly emphasizes that detecting covert aggression (CAG) is significantly more difficult than overt aggression (OAG). This is attributed to the subtle nature of CAG, often employing polite expressions insincerely or relying heavily on context.
- Context is King (and Missing): The authors repeatedly highlight that the lack of contextual information (e.g., user profiles, conversation threads) severely hinders the accurate detection of aggression, especially covert forms. Understanding the "real intention" often requires knowing the reader's and writer's perspectives.
- Feature Engineering Matters, but Not Always More is Better: While combining various features (textual, sentiment, n-grams) generally improved performance, adding more features did not always lead to better results. Some features, like emojis or specific POS tags, could introduce noise and degrade performance, suggesting a need for careful feature selection.
- Simple Features Can Be Powerful: For traditional classification techniques, simple textual features like TF-IDF demonstrated comparable descriptive power to more complex features, suggesting that for certain tasks, a simpler approach might suffice.
- Platform Differences are Significant: The performance of the models varied between Facebook and Twitter tasks, underscoring the impact of platform-specific characteristics and language usage on aggression detection.
Key Results
- Facebook vs. Twitter Performance: The proposed models generally performed better on the Facebook task compared to the Twitter task.
- OAG vs. CAG Detection: The models achieved higher True Positive Rates (TPR) for overt aggression (OAG) and non-aggression (NAG) but struggled significantly with covert aggression (CAG), with TPRs as low as 23.97% for the Facebook task.
- Model Performance: The "Full Model" (averaging the Neural Network and SVM) achieved an F1-weighted score of 0.5948 for the Facebook task and 0.5480 for the Twitter task in the shared challenge.
- Feature Set Performance: For the Kumar et al. dataset, TF-IDF+SentiWordNet performed best during the design phase, while StanfordSentiment performed worst. However, when tested on the Facebook test set, StanfordSentiment showed the best results, indicating that validation set performance doesn't always predict test set performance.
- Statistical Significance: Wilcoxon tests revealed that for the Kumar et al. dataset, many feature sets did not show statistically significant differences in performance. For the Reynolds et al. dataset, no statistical differences were observed across feature sets.
Limitations and Future Work
Limitations: * Lack of Context: The most significant limitation is the absence of user profile information and post context, which are crucial for understanding intent, especially for covert aggression. * English Specificity: The current approach is primarily designed for English text, limiting its applicability to other languages. * Dataset Bias: The Kumar et al. dataset, gathered from Indian Facebook and Twitter pages, might contain idiomatic expressions and colloquialisms that differ from standard English, potentially affecting model generalization.
Future Work: * Incorporate Contextual Information: Exploring user profiles, common expressions, and general user behavior could significantly improve detection accuracy. * Explore Alternative Architectures: Investigating other deep learning architectures like Convolutional Neural Networks (CNNs) or Bidirectional LSTMs (BiLSTMs) could be beneficial. * Unsupervised Learning: Developing unsupervised machine learning algorithms for feature generation could help extend the model to different languages without requiring new corpora. * Adaptation to Evolving Language: Researching methods to adapt the model to changes in language usage and new forms of aggressive behavior is essential.
Summary updated