TL;DR
This paper tackles the challenge of classifying "troll memes" in Tamil-English code-mixed text, a common form of online communication. By comparing various classical and pre-trained language models, they found that the MuRIL model, specifically trained on Indian languages, achieved the best performance, highlighting the importance of language-specific embeddings for this complex task.
What Problem Does This Paper Address?
The internet has become a primary mode of communication, with social media platforms playing a significant role. This has led to a rise in "trolling," defined as intentionally provoking conflict online. A particular challenge in understanding online discourse, especially in non-English speaking regions, is the prevalence of "code-mixed" text. This refers to the mixing of native languages (like Tamil) with English, often written using the Roman script due to keyboard limitations and the fast pace of online interaction. This paper focuses on classifying "troll memes" within this Tamil-English code-mixed context, aiming to automatically identify content intended to provoke or instigate conflict. The inherent ambiguity in syntax and semantics, coupled with the informal nature of social media text (spelling mistakes, slang, phonetic typing), makes this a difficult problem for machines to solve.
Methods
The authors explored two main approaches for generating text embeddings:
- Raw Code-Mixed Text: They directly processed the Tamil-English code-mixed captions. Techniques used here included:
- TF-IDF: A classic term frequency-inverse document frequency approach, with n-gram ranges tuned for better performance.
- Sub-word LSTM: Long Short-Term Memory networks that process text at the sub-word level, capturing sequential information.
- m-BERT: A pre-trained BERT model adapted for multilingual text.
- Translated and Transliterated Text: They created versions of the text where English words were translated to Tamil, and Tamil words written in Roman script were transliterated into the Tamil script. Embeddings were then generated from these processed texts using:
- TF-IDF
- IndicFT: A fastText model trained on Indian languages.
- MuRIL: A pre-trained multilingual model specifically designed for Indian languages.
- m-BERT
These embeddings were then fed into eleven different classification algorithms, including Logistic Regression, Support Vector Machines (with various kernels), Random Forest, k-Nearest Neighbors, Extra-tree classifier, AdaBoost, XGBoost, and Multilayer Perceptron. A voting ensemble of these classifiers was also used.
Datasets
The study utilized the official dataset released for the DravidianLangTech-ACL2022 challenge. This dataset consists of meme captions labeled as either "troll" or "not troll." The data is presented in a Tamil-English code-mixed format, with sentences containing both Tamil and English words written in Roman script.
- Training Data: 2300 captions (1282 troll, 1018 not troll).
- Test Data: 667 captions (395 troll, 272 not troll).
Key Insights and Ideas
- The Power of Language-Specific Pre-training: The most significant finding is that pre-trained models specifically designed for Indian languages, like MuRIL, significantly outperform general multilingual models (like m-BERT) and classical methods when dealing with code-mixed Dravidian languages. This suggests that capturing the nuances of these languages is crucial for effective classification.
- Translation/Transliteration Trade-offs: While the authors hypothesized that translation and transliteration would improve accuracy, the results showed mixed outcomes. For TF-IDF, direct processing of raw code-mixed text performed better. However, for pre-trained models like MuRIL, translation and transliteration, when combined with the model's specialized training, led to superior results. This indicates that the effectiveness of these preprocessing steps is highly dependent on the downstream model.
- Code-Mixed Text is a Unique Challenge: The study underscores the complexity of code-mixed text. The informal nature, spelling variations, and linguistic mixing require specialized approaches rather than simply applying standard NLP techniques.
Key Results
The paper reports the following key results:
- The MuRIL model, when applied to translated and transliterated text, achieved the highest weighted F1 score of 0.74. This significantly beat their previously published competition result of 0.60.
- IndicFT achieved a weighted F1 score of 0.70 on translated and transliterated text.
- m-BERT performed slightly better with translated-transliterated text (F1 score of 0.55) compared to raw text (0.53).
- For TF-IDF, processing the raw code-mixed text yielded better results across accuracy, macro F1, and weighted F1 scores compared to using translated and transliterated text.
Limitations and Future Work
- Limitations: The authors acknowledge that their study focused solely on text captions and did not incorporate the meme images themselves. They also noted that for TF-IDF, translation and transliteration performed poorly, which they attributed to lower confidence in the capabilities of those specific translation/transliteration tools for code-mixed text.
- Future Work: The authors propose developing a unified framework that combines both the text captions and the meme images for classification. They also suggest integrating deep learning models that can process both modalities simultaneously to potentially achieve even better results.
Summary updated