TL;DR
This paper introduces a multimodal approach for detecting hate speech in Dravidian languages (Tamil, Malayalam, Telugu) by combining text and audio data. Their hybrid model, using language-specific BERT for text and MFCC for audio, achieved top rankings in a recent shared task, demonstrating the effectiveness of multimodal analysis for under-resourced languages.
What Problem Does This Paper Address?
The proliferation of social media has unfortunately led to an increase in online hate speech. Detecting this harmful content is crucial for maintaining safe digital spaces. However, existing research often focuses on text-only analysis and primarily on well-resourced languages. This paper tackles the challenge of detecting hate speech in Dravidian languages (Tamil, Malayalam, and Telugu), which are considered under-resourced in NLP research. A key complexity is the rise of multimodal content, where hate speech can be conveyed through a combination of text, speech, and other media, making detection more difficult, especially when considering the linguistic diversity and phonetic variations inherent in these languages.
Methods
The authors propose a hybrid approach that leverages both text and audio modalities. For the text component, they utilize language-specific BERT models (Tamil BERT, Malayalam BERT, Telugu BERT) to generate contextualized word embeddings. These embeddings serve as rich feature representations. For the audio component, they extract features using Mel-Frequency Cepstral Coefficients (MFCC) and spectral representations, which capture key auditory information like tone and intonation.
These multimodal features are then combined. Before classification, the dataset undergoes preprocessing, including text tokenization and noise reduction for audio. To address class imbalance, the Synthetic Minority Over-sampling Technique (SMOTE) is employed. Finally, a Random Forest classifier is used for the multimodal classification task. The authors chose Random Forest due to its ability to handle high-dimensional feature spaces, perform implicit feature selection, and its robustness compared to other models like SVM.
Datasets
The study utilizes data from the Multimodal Social Media Data Analysis in Dravidian Languages (MSMDA-DL) shared task at DravidianLangTech@NAACL 2025. The dataset consists of text and speech components sourced from social media platforms for Tamil, Malayalam, and Telugu. The training datasets contain 514 (Tamil), 883 (Malayalam), and 556 (Telugu) rows, while the test datasets for each language consist of 50 rows. Each data sample includes transcribed text (handling code-mixing and informal language) and corresponding audio samples. The dataset is annotated for hate speech classification, with a "Hate" class further divided into subclasses (Gender, Political, Religious, Personal Defamation) and a "Non-Hate" class.
Key Insights and Ideas
- Multimodal Advantage: Combining text and audio significantly enhances hate speech detection, especially for subtle forms of hate speech that rely on non-verbal cues.
- Language-Specific BERT Power: Using language-specific BERT models for text processing in Dravidian languages proves highly effective, capturing nuanced linguistic features better than generic models.
- Random Forest for Fusion: Random Forest emerges as a strong classifier for multimodal data, effectively handling the complexity and high dimensionality of combined features.
- Under-Resourced Language Potential: The success in Tamil and Malayalam demonstrates that even with limited resources, multimodal approaches can achieve competitive performance.
- Telugu as a Challenge: The lower performance in Telugu highlights the persistent challenges posed by linguistic variations, phonetics, and potentially less robust data for certain under-resourced languages.
Key Results
The proposed multimodal approach achieved impressive results in the MSMDA-DL shared task:
- Tamil: Macro-F1 score of 0.7332 (Rank 1)
- Malayalam: Macro-F1 score of 0.7511 (Rank 1)
- Telugu: Macro-F1 score of 0.3758 (Rank 2)
These scores indicate that the model was able to effectively balance precision and recall, particularly for Tamil and Malayalam. The confusion matrices for Tamil and Malayalam were notably similar, suggesting consistent performance across these two languages.
Limitations and Future Work
The authors acknowledge several limitations:
- Data Dependency: The model's effectiveness is heavily reliant on the availability and quality of labeled datasets, which are scarce for many low-resource languages.
- Language-Specific BERT Bias: Reliance on language-specific BERT models might introduce biases if the pre-training data doesn't adequately represent dialectal variations or informal speech.
- Audio Feature Challenges: Background noise, variations in pronunciation, and recording quality can impact audio feature extraction (e.g., MFCCs), leading to misclassifications.
- Computational Resources: Multimodal models require higher computational power, posing challenges for real-time deployment and scalability in resource-constrained environments.
- Feature Alignment Complexity: Aligning features from asynchronous or incomplete audio and text inputs adds complexity.
- Interpretability: Transformer-based models and deep learning approaches often act as "black boxes," making it difficult to explain classification decisions.
Future work suggestions include:
- Domain-Specific Fine-Tuning: Training models on data specific to particular domains or social media platforms to improve context-awareness.
- Advanced Multi-Modal Fusion: Exploring more sophisticated fusion strategies (early, late, or hybrid) to better integrate text and audio features.
- Improved Data Augmentation: Developing techniques to enhance data quality and quantity for under-resourced languages.
- Noise-Robust Feature Extraction: Investigating methods to make audio feature extraction more resilient to noise.
- Explainability Techniques: Incorporating methods to provide clearer explanations for model decisions.
Summary updated