An Ensemble Approach for Aggression Identification in English and Hindi Text

Arjun Roy, Prashant Kapil, Kingshuk Basak, Asif EkbalCOLING 2018

TL;DR

This paper presents an ensemble system for identifying aggression in online text, tackling the challenge of code-mixed languages (English and Hindi). By combining Convolutional Neural Networks (CNNs) and Support Vector Machines (SVMs), the system achieves competitive performance on a challenging task with real-world implications for online safety.

What Problem Does This Paper Address?

The proliferation of social media has unfortunately led to an increase in online aggression, cyberbullying, and hate speech. Identifying these harmful behaviors is crucial for creating safer online environments. However, this task is complicated by several factors:

  • Overlapping Characteristics: Aggression, cyberbullying, and hate speech often share similar linguistic features, making them difficult to distinguish.
  • Code-Mixing: Online communication, especially in multilingual contexts like India, frequently involves mixing languages within a single sentence or post. This "code-mixing" poses a significant challenge for traditional NLP models trained on single languages.
  • Nuance and Indirectness: Aggression can be overt (direct insults) or covert (sarcasm, subtle digs), making it hard to detect subtle forms of harm.

This paper aims to address these challenges by developing a robust system for aggression identification in both English and Hindi, specifically designed to handle code-mixed text.

Methods

The authors propose an ensemble approach that combines two powerful machine learning models:

  • Convolutional Neural Networks (CNNs): CNNs are well-suited for text classification as they can effectively capture local patterns (like n-grams) within the text. The authors use a CNN with a specific filter size and ReLU activation, followed by max-pooling to extract features.
  • Support Vector Machines (SVMs): SVMs are a classic and effective algorithm for classification. The authors use an SVM trained with John Platt's sequential minimal optimization algorithm.

The outputs from the CNN and SVM models are concatenated and fed into a final dense layer with a dropout layer and a Softmax activation function for classification. This ensemble strategy aims to leverage the strengths of both individual models to improve overall performance.

Preprocessing is a critical step, involving:

  • Removal of URLs.
  • Selective removal of '#' symbols to preserve meaning.
  • Removal of punctuation and apostrophe words.
  • Crucially, for code-mixed Hindi text, transliterated Hindi words and actual English words are converted to actual Hindi words using the Indic-Trans API.

Word embeddings are used to represent words numerically:

  • For Hindi, a pre-trained model of Wikipedia is used.
  • For English, pre-trained GloVe vectors are employed. Sentences are padded or pruned to a maximum length of 50 words.

Features used for the SVM model include:

  • Unigrams: Top 1000 unigrams from the corpus.
  • TF-IDF vectors: Term Frequency-Inverse Document Frequency to weigh word importance.

Datasets

The study utilizes datasets from the COLING 2018 TRAC-1 shared task on Aggression Identification.

  • Training Data:
    • Hindi: 15,000 annotated instances (OAG, CAG, NAG).
  • English: 12,000 annotated instances (OAG, CAG, NAG) from Facebook posts and comments.
  • Validation Data:
    • Hindi: 3,000 annotated instances.
  • English: 3,000 annotated instances.
  • Test Data: Two sets were used for evaluation:
  • Facebook data (English and Hindi).
  • Other social media data (English and Hindi).

Key Insights and Ideas

  • Ensemble Power: Combining CNN and SVM models proved more effective than individual models, highlighting the benefit of ensemble methods for complex NLP tasks.
  • Code-Mixing is a Major Hurdle: The authors explicitly acknowledge the difficulty of code-mixed text and the necessity of specific preprocessing steps (like Indic-Trans) to handle it. This underscores the need for language-aware NLP solutions in diverse linguistic environments.
  • Overt vs. Covert Aggression Detection: The paper highlights that detecting overt aggression is generally easier than covert aggression, which relies on subtle cues and sarcasm. This is reflected in their error analysis, where OAG is often confused with CAG, and CAG with NAG.
  • Importance of Preprocessing: The detailed preprocessing steps, especially for code-mixed text, demonstrate that careful data preparation is as crucial as the model architecture itself for achieving good performance.

Key Results

The ensemble system achieved competitive F1 scores on the test data:

  • English Facebook: F1 score of 0.5151
  • English Social Media: F1 score of 0.5099
  • Hindi Facebook: F1 score of 0.5599
  • Hindi Social Media: F1 score of 0.3790

The results show that the system performed better on English data and Hindi Facebook data compared to Hindi social media data. The error analysis revealed that OAG is often confused with CAG, and CAG with NAG, due to shared keywords and indirect language.

Limitations and Future Work

  • LSTM Performance: The authors note that an LSTM model did not perform as well as expected in their experiments, suggesting that exploring attention mechanisms with LSTMs could be beneficial.
  • NER for Hashtags: The paper suggests that extracting Named Entity Recognition (NER) features, especially from hashtags which are common in aggressive posts, could significantly improve performance.
  • Hybrid Architectures: The authors express interest in exploring other relevant linguistic features and hybrid machine learning architectures to further enhance the system's capabilities.
  • Code-Mixed Hindi Performance: The lower performance on Hindi social media data, likely due to more complex code-mixing and informal language, indicates a need for further research in this specific area.

Summary updated

BibTeX
@inproceedings{roy2018ensemble,
  author = {Arjun Roy and Prashant Kapil and Kingshuk Basak and Asif Ekbal},
  title = {{An Ensemble Approach for Aggression Identification in English and Hindi Text}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure