TL;DR
This paper presents a machine learning approach for identifying aggressive language on social media, combining word embeddings and sentiment features. The study found that while Random Forests performed better on domain-specific data (Facebook), Support Vector Machines were more robust for out-of-domain data (Twitter), highlighting the importance of feature engineering and model choice for social media text analysis.
What Problem Does This Paper Address?
Social media platforms have become a primary communication channel, but they also host a significant amount of aggressive and abusive content. Manually moderating this content is infeasible due to the sheer volume of data generated daily. This necessitates the development of automatic methods for identifying and flagging such content. The paper addresses the challenge of accurately classifying social media texts into categories of overt aggression, covert aggression, and non-aggression, a task that is particularly difficult due to the nuances of online communication and the potential for ambiguity in language.
Methods
The authors employed a machine learning approach that leverages three main sources of information for feature extraction:
- Word Embeddings: They utilized pre-trained GloVe vectors (300-dimensional) to represent words. For each text, the word vectors were averaged to create a single vector representation. Words not found in the GloVe vocabulary were ignored.
- Emoticons: Emoticons were treated as indicators of sentiment and aggression. A TF*IDF-like score was calculated for each emoticon with respect to each aggression class, based on their frequency within those classes. The final emoticon score for a text was the sum of these scores for emoticons present in the text, normalized by the number of emoticons.
- Sentiment Features: Sentiment analysis was performed using SentiStrength, which provides scores for both positive and negative sentiment in a text. The intuition here is that aggressive texts are more likely to be negative. These features were not used in all experimental runs.
Two machine learning classifiers were evaluated: Support Vector Machines (SVMs) with a radial basis function (RBF) kernel, and Random Forests. The parameters for both models were tuned using a grid search on a development dataset.
Datasets
The research utilized data from the First Shared Task on Aggression Identification. This task provided:
- Training Dataset: Containing texts and comments from Facebook.
- Test Datasets:
- A Facebook dataset, similar to the training data.
- A Twitter dataset, which was considered out-of-domain compared to the training data.
The evaluation metric used was the weighted macro-averaged F-measure.
Key Insights and Ideas
- Domain Sensitivity: The study revealed a significant difference in performance between the Facebook (in-domain) and Twitter (out-of-domain) datasets. Random Forests excelled on the Facebook data, while SVMs performed better on the Twitter data. This suggests that models trained on one social media platform may not generalize well to another without adaptation.
- Feature Importance: The inclusion of sentiment features consistently improved the performance of both SVM and Random Forest models, underscoring the role of emotional tone in identifying aggression.
- Simplicity vs. Sophistication: Despite using relatively simple features (averaged word embeddings, emoticons, and sentiment scores) and standard machine learning algorithms, the proposed method achieved competitive results, outperforming more complex approaches in some cases.
- Ambiguity of Emoticons: The analysis of emoticons showed that some, like ":face with tears of joy:", appeared across multiple aggression categories, highlighting the inherent ambiguity in interpreting emoticons in isolation.
Key Results
- Facebook Dataset: The best performing system on the Facebook dataset was RF-with-sentiment, achieving an F1 (weighted) score of 0.5830 and an accuracy of 0.5579. This system ranked 13th out of 30 participants.
- Twitter Dataset: The best performing system on the Twitter dataset was SVM-with-sentiment, achieving an F1 (weighted) score of 0.5074 and an accuracy of 0.5529. This system ranked 18th out of 30 participants.
- Class-Specific Performance: On the Facebook data, the OAG (Overtly Aggressive) class was the most problematic to identify, with a low F-score. On the Twitter data, the CAG (Covertly Aggressive) class had the lowest F-score.
- NAG Recall vs. Precision: For the Twitter data, the system achieved very high recall (>97%) for the NAG (Non-aggressive) class but lower precision (56%), indicating it correctly identified most non-aggressive texts but also misclassified some aggressive texts as non-aggressive.
Limitations and Future Work
- Tokenization: The paper acknowledges that the NLTK tokenizer used was not robust enough for all data, particularly Twitter data, and suggests that a more accurate tokenizer could improve results.
- Domain-Specific Embeddings: The authors did not experiment with word embeddings trained specifically on Twitter data, which they believe could lead to better performance on Twitter-based tasks.
- Model Architectures: The paper suggests that more powerful learning architectures, such as neural networks, and more informative features could significantly improve performance.
- Ambiguity of Covert Aggression: The difficulty in identifying covert aggression (CAG) was noted as a persistent challenge.
Summary updated