TL;DR
This paper presents LexiLogic's approach to detecting hate speech in Dravidian languages (Malayalam, Tamil, Telugu) for the DravidianLangTech@NAACL 2025 challenge. They achieved impressive results, including first place for Telugu and second for Malayalam and Tamil, by solely focusing on text-based analysis using fine-tuned, language-specific BERT models, demonstrating the power of text alone even in a multimodal task.
What Problem Does This Paper Address?
The proliferation of social media has led to an explosion of content in various languages, including Dravidian languages like Malayalam, Tamil, and Telugu. This surge necessitates robust systems for detecting harmful content, such as hate speech, to maintain online safety and civility. However, these languages present unique challenges due to their rich morphology, distinct scripts, and the prevalence of code-mixing (blending multiple languages within a single utterance). The DravidianLangTech@NAACL 2025 shared task specifically aimed to address this by providing a multimodal dataset (text and audio) for hate speech detection in these languages. The core problem this paper tackles is how to effectively detect hate speech in these linguistically complex, low-resource languages, especially when faced with imbalanced datasets.
Methods
The LexiLogic team opted for a focused, text-only approach, leveraging the power of pre-trained language models. Their core methodology involved:
- Language-Specific BERT Models: Instead of using a general multilingual BERT, they utilized BERT models specifically pre-trained for Malayalam, Tamil, and Telugu from the
13cube-punerepository. This choice is crucial as these models are better equipped to capture the nuances of each individual language. - Fine-tuning: These pre-trained models were then fine-tuned on the task-specific dataset for hate speech classification.
- Text-Only Analysis: Notably, they chose to exclude the audio data provided in the multimodal task, demonstrating that text features alone could yield competitive results.
- Parameter Optimization: They experimented with various training strategies, including label smoothing and weight decay regularization, to combat class imbalance. They also employed early stopping and model checkpointing to optimize performance and retain the best models.
- Hyperparameter Tuning: Key hyperparameters like learning rate (2e-5) and batch size (8) were systematically tuned, with slight adjustments to weight decay for Telugu.
Datasets
The primary dataset used was provided by the DravidianLangTech shared task at NAACL 2025. This dataset included:
- Languages: Malayalam, Tamil, and Telugu.
- Content: Text transcripts and audio recordings of social media content.
- Categorization: Content was categorized into "hate speech" and "non-hate speech," with hate speech further subdivided into four categories: Gender, Political, Religious, and Character.
A key characteristic of this dataset, as highlighted by the authors, is class imbalance, with the "Non-Hate" category being dominant across all languages. The distribution of hate speech subcategories also varied significantly, with "Character Defamation" being particularly prevalent in Malayalam and Telugu.
Key Insights and Ideas
- Text is Powerful: The most significant insight is that even in a multimodal task, a well-tuned, language-specific BERT model focusing solely on text can achieve state-of-the-art results. This challenges the assumption that multimodal fusion is always necessary for optimal performance.
- Language-Specific Models Matter: Using BERT models pre-trained specifically for Dravidian languages proved more effective than general multilingual models, underscoring the importance of linguistic specificity.
- Class Imbalance is a Major Hurdle: The authors explicitly discuss the challenges posed by class imbalance, noting that models tend to favor majority classes. This necessitates careful regularization and potentially other techniques to ensure fair detection across all categories.
- Development vs. Test Set Discrepancies: The paper highlights substantial performance gaps between their development set experiments and the final test set results, particularly for Telugu. This suggests that the test set contained more challenging or diverse examples, pointing to potential overfitting issues or generalization challenges.
- Code-Mixing Remains Tricky: Despite using language-specific models, the paper acknowledges that code-mixed content, common in Dravidian languages, still presents difficulties for optimal processing.
Key Results
The LexiLogic system achieved highly competitive results in the DravidianLangTech@NAACL 2025 shared task:
- Telugu: First place with a Macro F1 score of 0.3817.
- Malayalam: Second place with a Macro F1 score of 0.7367.
- Tamil: Second place with a Macro F1 score of 0.7225.
These results are particularly impressive given the text-only approach and the inherent challenges of the dataset, such as class imbalance. The performance gap between their development set (e.g., 0.90 Macro F1 for Telugu) and the final test set (0.3817 for Telugu) was significant, indicating that even with their strong performance, generalization to unseen data remains a critical area.
Limitations and Future Work
The authors acknowledge several limitations and propose future directions:
- Text-Only Reliance: The exclusion of audio data means potential missed paralinguistic cues (tone, emotion) that could aid in ambiguous cases.
- Computational Resources: Language-specific BERT models are computationally intensive, potentially limiting real-time applications.
- Class Imbalance: Despite regularization, models still favor majority classes, and minority hate speech categories remain challenging.
- Code-Mixed Content: Handling text that switches between English and Dravidian languages, or different scripts, is still an area for improvement.
- Generalization: The performance disparity between development and test sets highlights limitations in generalizing to new data patterns.
Future Work Suggestions:
- Multimodal Fusion: Integrating audio features using advanced speech encoders (e.g., IndicWav2Vec) could enhance performance.
- Data Augmentation: Implementing language-specific data augmentation techniques.
- Cross-Validation: Adopting more rigorous cross-validation strategies.
- Linguistic Features: Incorporating deeper linguistic analysis and script handling for better understanding of regional variations.
- Larger Models: Exploring newer, larger language models trained on Dravidian languages.
- Robust Validation: Developing more robust validation and adaptation strategies to handle data shifts.
Summary updated