TL;DR
This paper presents the team IIT(ISM)@COLING'18's approach to identifying aggression in social media, classifying it into Overtly Aggressive (OAG), Covertly Aggressive (CAG), and Non-aggressive (NAG) categories. While their Long Short-Term Memory (LSTM) model achieved modest results, falling short of baselines in one task, the work highlights the inherent difficulty in distinguishing subtle forms of aggression and sets a foundation for future research.
What Problem Does This Paper Address?
The proliferation of social media has led to a surge in online aggression, cyberbullying, and hate speech, posing significant social and even criminal challenges. Identifying these behaviors, especially the more nuanced forms like "covert aggression," is a complex task. This paper tackles the challenge of accurately classifying social media text into three distinct categories of aggression: overtly aggressive, covertly aggressive, and non-aggressive. This three-way classification is crucial for developing effective tools to combat online toxicity.
Methods
The team employed a Long Short-Term Memory (LSTM) model, a recurrent neural network architecture known for its effectiveness in sequence modeling tasks like natural language processing. For word representation, they utilized pre-trained embeddings from the DeepMoji model. A key aspect of their methodology was the "chain-thaw" transfer learning approach, which involves sequentially unfreezing and fine-tuning individual layers of the pre-trained model. This technique aims to adapt the model to the specific task while mitigating the risk of overfitting. The model architecture includes an embedding layer, a bidirectional LSTM layer for context capture, global max pooling for higher-level feature extraction, a ReLU activation layer, dropout for regularization, and a softmax layer for the final classification into the three aggression categories.
Datasets
The study utilized the Aggression Identification dataset, which was compiled for the TRAC-1 Shared Task. This dataset consists of 15,000 annotated Facebook posts and comments in both Hindi (Roman and Devanagari scripts) and English. The data was used for training and validation, with the goal of performing a three-way classification between Overtly Aggressive, Covertly Aggressive, and Non-aggressive text.
Key Insights and Ideas
- The subtle nature of covert aggression is a significant challenge: The paper repeatedly emphasizes that distinguishing between overt and covert aggression is difficult, even for humans. This inherent ambiguity makes it a particularly hard problem for machine learning models.
- Emojis as proxies for emotion: The authors hypothesize that emojis can serve as indicators of emotional content in text, suggesting that pre-training on emoji prediction tasks could improve aggression identification.
- Transfer learning with careful fine-tuning is crucial: The "chain-thaw" method, which incrementally fine-tunes model layers, is presented as a strategy to adapt pre-trained embeddings to the specific nuances of aggression detection without risking catastrophic forgetting or overfitting.
Key Results
The team participated in two tasks: English (Facebook) and English (Social Media).
- English (Facebook) Task: Their best run, LSTM-05, achieved a weighted F1 score of 0.3572, which was only slightly better than the random baseline (0.3535). Other runs (LSTM-01, LSTM-04) performed worse.
- English (Social Media) Task: Their system (LSTM-02) achieved a weighted F1 score of 0.1960, which was significantly lower than the random baseline (0.3477).
Confusion matrices revealed that the model was best at correctly predicting "NAG" comments. "CAG" comments were predicted with moderate accuracy, while "OAG" comments were the most challenging to classify, often being confused with "CAG." The English (Social Media) task showed an even larger discrepancy between "OAG" and "CAG" predictions.
Limitations and Future Work
The authors acknowledge that their overall performance was "dismal." They note that their DeepMoji embeddings, while useful, may not have fully captured the sub-categories of aggression. For future work, they suggest exploring other models such as Support Vector Machines (SVM) and ensemble models. Additionally, they propose investigating character-level Convolutional Neural Networks (CNNs) for extracting character-level features, followed by a Random Forest classifier, as a promising avenue for further research.
Summary updated