Abstract
This paper details the participation of the RCLN team in the Automatic Misogyny Identification (AMI) task at EVALITA 2018. The proposed system, "Tweetaneuse," explores language-independent, character-based methodologies for detecting misogynistic content in English and Italian tweets. By focusing on sub-word patterns rather than lexical dictionaries, the system aims to handle the inherent noise of social media text. Experimental results demonstrate that a relatively simple Random Forest model utilizing weighted character n-grams can outperform more complex deep learning architectures in data-constrained scenarios.
Introduction
The proliferation of hate speech on social media platforms, particularly misogyny targeting women, presents a significant challenge for automated moderation. Twitter data is notoriously "noisy," characterized by non-standard orthography, abbreviations, hashtags, and slang. Traditional Natural Language Processing (NLP) systems often rely on extensive language-specific lexical resources, which are difficult to maintain and scale across different languages.
The "Tweetaneuse" system addresses these challenges by employing character-based models. By analyzing text at the level of character sequences (n-grams), the system remains largely language-independent and robust to the creative misspellings common in online discourse. This study evaluates two primary approaches: a locally-weighted Bag-of-Ngrams model and a character/word-level Bi-LSTM network.
Methodology
The researchers investigated two distinct modeling paradigms to determine the most effective way to capture misogynistic patterns.
Locally-weighted Bag-of-Ngrams (lw RF)
The primary submission utilized a Random Forest (RF) classifier. Unlike standard Bag-of-Words models that simply count occurrences, this approach implemented a positional scoring scheme. The underlying hypothesis is that the location of a character sequence within a tweet—specifically at the beginning or end—carries significant semantic weight in short-form communication. Character n-grams ranging from size 3 to 6 were extracted and weighted based on their position before being fed into the Random Forest.
Bi-LSTM Architecture
As a more sophisticated alternative, the team developed a Bidirectional Long Short-Term Memory (Bi-LSTM) network. This model processes text hierarchically: first, a character-level layer builds representations of text fragments (words or sub-words); subsequently, a word-level layer processes these fragments to capture the overall context of the tweet. While theoretically more powerful, this model was primarily used for post-hoc analysis due to performance issues during the development phase.
Experimental Setup
The models were evaluated using the AMI dataset provided by EVALITA 2018. The dataset consisted of 4,000 training tweets and 1,000 test tweets for both English and Italian. The evaluation focused on two subtasks:
- Subtask A: Binary classification to identify whether a tweet is misogynistic.
- Subtask B: Multi-class classification to identify the specific behavior (e.g., discredit, dominance, sexual harassment) and the target (active/individual vs. passive/generic).
Results and Discussion
The character-based Random Forest model (lw RF) served as the primary entry. In Subtask A, the system achieved a Macro F1-score of 0.824 for Italian, indicating high effectiveness in capturing misogynistic patterns in that language. However, performance on English tweets was lower, with a Macro F1-score of 0.586. This discrepancy suggests that misogynistic expressions in English may be more linguistically diverse or less easily captured by the specific n-gram ranges used.
For Subtask B (behavior classification), performance dropped significantly (0.473 for Italian and 0.165 for English). This decline is largely attributed to extreme data imbalance; the "discredit" category dominated the training set, making it difficult for models to learn the characteristics of rarer behaviors like "dominance" or "derailing."
A key finding was the superiority of the "simpler" Random Forest model over the Bi-LSTM. The deep learning model suffered from overfitting, frequently defaulting to the majority class. This highlights a common pitfall in applying complex neural networks to relatively small datasets (4,000 samples), where feature engineering and traditional classifiers can often prove more robust.
Conclusion
The Tweetaneuse system demonstrates the viability of language-independent, character-based methods for misogyny identification. The success of the locally-weighted Random Forest model underscores the importance of positional information in short-form text analysis. Future research will focus on addressing data imbalance through oversampling or synthetic data generation and exploring the integration of word-sense disambiguation to complement character-level features.
This paper was originally presented at the EVALITA 2018 workshop as part of the AMI task.
Summary updated