TL;DR
This paper tackles the challenge of detecting abusive comments in Tamil, a low-resource language, by combining linguistic features with transformer-based sentence embeddings. Their novel knowledge integration strategy achieved a respectable 3rd place in the TamilNLP-ACL2022 shared task for code-mixed Tamil, highlighting the importance of multi-faceted approaches for nuanced language understanding.
What Problem Does This Paper Address?
Social media has unfortunately become a breeding ground for online harassment and abuse. Automatically detecting these harmful comments is crucial, but it's particularly challenging for languages like Tamil, which have fewer resources and less established state-of-the-art techniques compared to high-resource languages. This paper focuses on the TamilNLP-ACL2022 shared task, which aimed to classify abusive comments in Tamil, both in its native script and in a code-mixed format (Tamil written using Latin characters). The task involves identifying various categories of abuse, including misandry, misogyny, xenophobia, homophobia, and transphobia, alongside "hope-speech" and "none-of-the-above."
Methods
The UMUTeam employed a knowledge integration strategy to build their abusive comment detection system. This approach combines multiple feature sets to create a more robust model. Specifically, they extracted four types of features:
- Linguistic Features (LF): These are language-independent features derived from the UMU-TextStats tool, including stylometric measures (like word and sentence length, type-token ratio), emoji usage, and Part-of-Speech tags.
- Sentence Embeddings (SE): Non-contextual sentence embeddings from FastText were used. FastText has a pre-trained model for Tamil, and these embeddings are generated by averaging word embeddings within a sentence, ignoring word order.
- BERT Embeddings (BF): Contextual sentence embeddings from multilingual BERT were utilized.
- RoBERTa Embeddings (RF): Similarly, contextual sentence embeddings from XLM-RoBERTa were employed.
These individual feature sets were first used to train separate neural network models. Then, a knowledge integration strategy was applied, where a new neural network was trained with all these feature sets combined as input. This allowed the model to learn from the complementary strengths of each feature type. They also explored an ensemble learning approach, combining predictions from individual models, but ultimately favored the knowledge integration strategy.
Datasets
The TamilNLP-ACL2022 shared task provided two datasets:
- Tamil-script dataset: Comments written in the native Tamil script.
- Code-mixed dataset: Comments written in Tamil using Latin characters.
Both datasets were divided into training and development sets. The paper notes a significant imbalance in the number of examples across different labels in both datasets, with the "none-of-the-above" class being the majority. The code-mixed dataset was also considerably larger than the Tamil-script dataset.
Key Insights and Ideas
- The power of combining diverse features: The success in the code-mixed setting suggests that integrating linguistic cues with powerful transformer embeddings is a promising direction for low-resource language NLP tasks.
- Code-mixed data presents unique challenges and opportunities: The model performed significantly better on code-mixed Tamil, possibly due to its larger size and the inherent expressiveness of mixing languages.
- Label imbalance is a major hurdle: The authors explicitly mention the difficulty in accurately classifying under-represented labels, like "transphobic" in the code-mixed dataset, due to the limited data.
- Language and cultural dependency: Hate speech detection is inherently tied to cultural nuances, making it harder to generalize across languages.
Key Results
The UMUTeam achieved the following results:
- Code-mixed Tamil: Reached 3rd position in the official leaderboard with a macro-average F1-score of 35%.
- Tamil-script: Achieved 9th position in the official ranking.
- The paper reports a macro F1-score of 49.834% for Code-mixed and 46.167% for Tamil-script on their validation split using the knowledge integration strategy.
- The "none-of-the-above" label (the majority class) consistently yielded the best results across both datasets.
- Performance on specific labels varied significantly. For instance, the model achieved good precision for "xenophobia" in code-mixed Tamil (80.357%) but struggled in Tamil-script (48.936%). The "transphobic" label in Tamil-script showed promising results (66.667%), but its counterpart in code-mixed was significantly lower (24.561%).
Limitations and Future Work
- Limited performance on Tamil-script: The authors acknowledge the significantly lower performance on the Tamil-script dataset and hypothesize that the model might be predicting the same result consistently, indicating a failure to learn effectively.
- Data imbalance: The limited number of examples for certain abusive categories poses a significant challenge for accurate classification.
- Future Work: The team plans to focus on developing more language-independent linguistic features, adapting tools like UMUTextStats to utilize different Part-of-Speech models (e.g., from Stanza) to expand feature subsets for Tamil. They also aim to incorporate idioms and expand dictionaries to better handle figurative language, a known challenge in hate speech detection.
Summary updated