TL;DR
This paper tackles the challenging problem of identifying offensive language in code-mixed Dravidian languages (Tamil, Kannada, Malayalam) found on YouTube. By fine-tuning a pre-trained multilingual BERT model, the authors achieved competitive results, ranking third overall in a shared task and demonstrating BERT's effectiveness for low-resource, code-mixed text.
What Problem Does This Paper Address?
Social media has become a primary channel for communication, with users frequently mixing languages within their posts and comments – a phenomenon known as code-mixing. This is particularly prevalent in low-resource languages like Tamil, Kannada, and Malayalam, where users might blend their native tongue with English. Identifying offensive content within this code-mixed text is crucial for various applications, from content moderation to understanding online discourse. However, the scarcity of labeled datasets and the inherent complexity of code-mixed language pose significant challenges for traditional natural language processing (NLP) models, which are often trained on monolingual data. This paper aims to bridge this gap by developing an effective offensive language identification system for these Dravidian code-mixed languages.
Methods
The core of this research lies in leveraging the power of pre-trained transformer models. The authors adopted a bidirectional approach using BERT (Bidirectional Encoder Representations from Transformers), specifically the bert-base-multilingual-cased model. This model is pre-trained on a massive corpus of text from over 100 languages, allowing it to capture contextual relationships within sentences effectively.
The key technique employed is fine-tuning. This involves taking the pre-trained BERT model and further training it on the specific task of offensive language identification using the collected Dravidian code-mixed datasets. This transfer learning approach allows the model to adapt its general language understanding capabilities to the nuances of offensive language in the target languages.
Before feeding the data into BERT, two preprocessing steps were applied:
- Emoji Conversion: Emojis, which are common in social media, were converted into text representations to preserve their emotional context.
- Punctuation Removal: Punctuation marks were removed from the input text to simplify processing.
The BERT model's architecture involves an encoder that reads the input text and a decoder that makes predictions. For this task, the model was configured for text classification, outputting one of six predefined labels.
Datasets
The study utilized existing, manually annotated datasets for Tamil-English, Kannada-English, and Malayalam-English code-mixed content. These datasets were provided by organizers of the EACL 2021 shared task on Offensive Language Identification in Dravidian Languages. The paper notes that the Tamil dataset was significantly larger than the Kannada and Malayalam datasets. The data was split into training and validation sets with an 8:2 ratio for training.
Key Insights and Ideas
- BERT's Efficacy in Low-Resource Code-Mixing: The study strongly suggests that pre-trained multilingual BERT models, when fine-tuned, are highly effective for offensive language identification even in low-resource, code-mixed scenarios. This overcomes the limitations of traditional unidirectional models.
- The Challenge of Data Imbalance: Despite the larger size of the Tamil dataset, its F1-score was lower than Malayalam's. This highlights the potential impact of data imbalance and the need for strategies to address it, even when using powerful models.
- Importance of Contextual Understanding: The bidirectional nature of BERT allows it to understand context from both left-to-right and right-to-left, which is crucial for deciphering the often subtle nuances of offensive language in code-mixed text.
- Multilingual Models Outperform Monolingual for Code-Mixing: The success of the multilingual BERT model in this code-mixed task underscores the advantage of using models trained on diverse linguistic data when dealing with language mixing.
Key Results
The proposed BERT-based system achieved the following weighted average F1-scores on the test set:
- Malayalam: 0.96 (Ranked 2nd out of 30 participants)
- Tamil: 0.73 (Ranked 6th out of 31 participants)
- Kannada: 0.70 (Ranked 5th out of 29 participants)
Overall, the "Hypers" team ranked 3rd among all submitted systems in the shared task, demonstrating the model's strong performance across all three languages. The authors note that their F1-scores were very close to the top-ranked teams, indicating the competitiveness of their approach.
Limitations and Future Work
The authors acknowledge several limitations and propose future directions:
- Data Imbalance: The significant imbalance in dataset sizes across languages, particularly the smaller datasets for Kannada and Malayalam, might have impacted performance.
- Potential for Further Improvement: Despite good results, the authors believe performance can be further enhanced.
- Future Work:
- Character-Level Embeddings: Incorporating character-level embeddings alongside word embeddings to better represent out-of-vocabulary (OOV) words and misspelled terms.
- Handling Imbalanced Data: Employing oversampling techniques to increase the representation of minority classes in the training data.
- Exploring Other Approaches: Investigating alternative approaches to further boost model performance.
Summary updated