TL;DR
This paper tackles the challenge of detecting abusive content in Tamil, a language with a significant amount of code-switching (mixing Tamil and English). They propose using transliteration as a data augmentation technique to expand their training data, which significantly improved performance and led to a 3rd place finish in the DE-ABUSE@TamilNLP-ACL 2022 shared task.
What Problem Does This Paper Address?
The internet has become a powerful tool for communication, but it also presents a challenge: the prevalence of abusive content. This is particularly problematic for languages like Tamil, where users often mix Tamil and English in their online interactions (code-switching). This code-mixed nature makes it difficult for standard NLP models to accurately detect abuse. Existing datasets might be small or lack sufficient examples of this code-mixed language, hindering the development of robust abuse detection systems. The authors aim to address this data scarcity and the complexity of code-mixed Tamil by proposing a novel data augmentation strategy.
Methods
The core of their approach is transliteration as data augmentation. They take existing code-mixed Tamil-English sentences and use a Python package (ai4bharat-transliteration) to convert them into pure Tamil sentences while preserving their original meaning. This effectively increases the size and diversity of their training data.
For the actual abuse detection model, they experimented with two main categories:
- Traditional Machine Learning (ML) Models: They used a pipeline involving TF-IDF vectorization followed by models like LightGBM, Catboost, Random Forest, Support Vector Machines, and Multinomial Naive Bayes.
- Transformer Models: They specifically fine-tuned the MURIL model, a pre-trained BERT model designed for Indian languages. MURIL was chosen because it was trained on both translated and transliterated data, making it well-suited for this task.
Datasets
The authors utilized two datasets provided for the shared task:
- Tamil Dataset: Consisting of Tamil sentences.
- Code-mixed Tamil-English Dataset: Containing sentences that mix Tamil and English.
After preprocessing and transliteration, they combined these datasets to create a larger, more comprehensive training set. The paper mentions that the original Tamil dataset had 2,240 training and 560 validation sentences, while the code-mixed dataset had 5,948 training and 1,488 validation sentences. The combined dataset, after transliteration and filtering, reached approximately 8,186 sentences.
Key Insights and Ideas
- Transliteration is a powerful data augmentation tool: The most significant insight is that transliterating code-mixed data into the target language (Tamil) can effectively overcome data scarcity and improve model performance in abuse detection. This is a practical and efficient way to leverage existing resources.
- Transformer models excel in this domain: The results clearly indicate that transformer-based models, specifically MURIL, significantly outperform traditional ML models for this task. This highlights the continued dominance of deep learning architectures for complex NLP problems.
- Code-switching is a major hurdle: The paper implicitly emphasizes the difficulty of handling code-switched language, which is common in online communication for many languages, including Tamil. Their approach directly tackles this challenge.
Key Results
The authors achieved a 3rd place ranking in the DE-ABUSE@TamilNLP-ACL 2022 shared task. Their best performing model, MURIL, achieved:
- Macro average F1 score of 0.290 on the test dataset.
- Weighted F1 score of 0.590 on the test dataset.
They also reported that the fine-tuned MURIL model outperformed all other ML models they experimented with. For traditional ML models, LightGBM showed the best performance with a macro average F1-score of 0.32 and a weighted average F1-score of 0.65 before hyperparameter tuning.
Limitations and Future Work
- Limitations: The paper doesn't explicitly detail many limitations. However, it's implied that the initial dataset size for pure Tamil was a limitation, which they addressed with transliteration. The performance metrics, while good enough for 3rd place, suggest there's still room for improvement in accurately detecting all forms of abuse.
- Future Work: The paper concludes by stating that Transformer models outperform traditional ML models for this task, suggesting that further exploration and fine-tuning of transformer architectures would be a promising direction. They also implicitly suggest that expanding the dataset further, perhaps through more sophisticated augmentation techniques or by collecting more diverse code-mixed data, could lead to even better results.
Summary updated