TL;DR
This paper tackles the challenge of automatically detecting aggressive content in social media. The authors demonstrate that deep neural networks, particularly an LSTM model, can achieve state-of-the-art performance when augmented with data augmentation and pseudo-labeling techniques, outperforming traditional methods.
What Problem Does This Paper Address?
The proliferation of online social platforms has unfortunately led to an increase in anti-social behaviors like trolling, flaming, abuse, and hate speech. Detecting these behaviors automatically is crucial for fostering a more civil online environment. However, social media text is often noisy, containing misspellings, abbreviations, and code-mixing, making traditional Natural Language Processing (NLP) feature extraction difficult and error-prone. Furthermore, aggressive content is often a minority class in datasets, leading to imbalanced data challenges. This paper aims to address these issues by exploring the effectiveness of deep neural networks for aggression detection, specifically investigating how data augmentation and pseudo-labeling can improve model performance, especially in the context of limited or imbalanced data.
Methods
The researchers experimented with a range of models, starting with a strong linear baseline: a Support Vector Machine (SVM) using naive Bayes log-count ratio features (NBSVM) with character n-grams. They then explored various deep neural network (DNN) architectures, including Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, Bidirectional LSTMs (BiLSTMs), and combinations thereof (CNN-LSTM, LSTM-CNN, CNN-BiLSTM, BiLSTM-CNN).
Key methodological innovations include:
- Data Augmentation: Inspired by computer vision techniques, they simulated label-preserving transformations by translating text into four intermediate languages (French, Spanish, German, Hindi) and then back to English. This process increased the training data size by a factor of 5.
- Pseudo-Labeling: To further diversify the training data, they used a model trained on the primary dataset to label data from an undisclosed social media platform (and publicly available Twitter data for hate speech detection). These pseudo-labeled examples were then added to the training set.
- Input Representation: They used a concatenation of 300-dimensional word vectors from fastText (leveraging sub-word information for better coverage) and a 50-dimensional task-specific word vector.
Datasets
The primary datasets used were:
- Facebook Dataset: Provided by the Aggression Detection in Social Media Shared Task, consisting of English and Hindi posts related to entities in India. The English portion was used, with 12,000 training, 3,001 development, and 916 test examples. This dataset exhibited class imbalance.
- Undisclosed Social Media Platform Dataset: 1,257 examples used for pseudo-labeling.
- Twitter Hate Speech Dataset: Publicly available data used for pseudo-labeling.
Key Insights and Ideas
- Data Augmentation is Key: The paper strongly suggests that data augmentation, particularly the translation-back-translation method, provides a significant performance boost for deep learning models, more so than pseudo-labeling alone. This highlights the importance of diverse lexical representations.
- Complexity Doesn't Always Mean Better: While more complex models like CNN-BiLSTM were explored, the simpler LSTM model, when trained with augmented data, achieved the best performance on the Facebook sub-task. This implies that for this specific task and dataset, a simpler architecture might generalize better.
- Noisy Text is a Real Challenge: The authors acknowledge that the inherent noise in social media text makes traditional NLP feature engineering difficult, motivating their exploration of deep learning models that can learn features automatically.
- Class Imbalance Matters: The performance trends across classes (better on NAG, worse on CAG) are linked to the distribution of examples in the training set, underscoring the ongoing challenge of handling imbalanced datasets in aggression detection.
Key Results
- Facebook Sub-task: The LSTM classifier, trained on augmented data, achieved a weighted macro-F1 score of 0.6425, ranking first overall out of 30 submissions.
- Social Media Sub-task: The CNN-LSTM model, trained with augmented and pseudo-labeled data, achieved a weighted macro-F1 score of 0.5920, ranking third overall out of 30 teams.
- Data Augmentation Impact: Data augmentation alone resulted in approximately a 5% weighted macro-F1 improvement on the development set over the NBSVM baseline.
- Model Performance: DNN models generally outperformed the NBSVM baseline, especially after data augmentation.
Limitations and Future Work
- Limitations:
- The paper notes that the translation-based data augmentation, while effective, is an "error-prone" process.
- The performance trend across classes (NAG > OAG > CAG) suggests that models struggle more with covertly aggressive (CAG) content, which is often more subtle.
- The authors acknowledge that a complex model doesn't always guarantee the best results, indicating that model selection and tuning are critical.
- Future Work:
- Investigating approaches to improve performance on classes with a minimal number of training examples (specifically, the CAG class).
- Exploring other data augmentation techniques beyond translation.
- Further research into combining different model architectures and feature representations.
Summary updated