TL;DR
This paper tackles the challenge of detecting hate speech in Dravidian languages (Tamil, Telugu, Malayalam) by proposing a multimodal approach that combines text and speech analysis. They introduce a late fusion technique using pre-trained models Muril (for text) and Wav2Vec 2.0 (for speech), demonstrating its effectiveness in improving detection accuracy, especially for low-resource languages.
What Problem Does This Paper Address?
Hate speech on social media is a growing concern, but automatically detecting it is particularly difficult in low-resource languages like Tamil, Telugu, and Malayalam. Traditional methods often focus solely on text, ignoring the rich information present in speech, especially on platforms with video and audio content. Dravidian languages present unique linguistic challenges, including complex morphology and syntax, which further complicate NLP tasks. Existing datasets are often imbalanced, with fewer examples of hate speech, and suffer from insufficient labeled data for training robust models. This paper aims to bridge this gap by developing a multimodal hate speech detection system that leverages both text and speech for these under-resourced languages.
Methods
The core of this work is a multimodal hate speech detection model that employs a late fusion technique. This means that the text and speech are processed independently by specialized models, and their predictions are combined at the end.
- Text Analysis: The authors use Muril, a transformer-based model pre-trained on 17 Indian languages. Muril is fine-tuned on the DravidianLangTech@NAACL 2025 dataset for text-based hate speech classification.
- Speech Analysis: For speech, they utilize Wav2Vec 2.0, a self-supervised model that learns speech representations directly from raw audio. This is particularly beneficial for low-resource languages as it doesn't require phonetic transcriptions and can handle dialectal variations better than traditional methods. Wav2Vec 2.0 is also fine-tuned on the speech data from the dataset.
- Late Fusion: The predictions (logits) from the Muril and Wav2Vec 2.0 models are combined using a weighted average. A hyperparameter
acontrols the contribution of each modality, allowing the model to balance their influence. This approach allows each modality to be optimized independently before aggregation, mitigating risks of feature redundancy and overfitting. - Addressing Class Imbalance: To combat the skewed distribution of hate speech categories, the researchers incorporate class weighting during training and employ data augmentation techniques for both text (synonym replacement, back-translation) and audio (Gaussian noise addition).
Datasets
The primary dataset used in this study is the DravidianLangTech@NAACL 2025 dataset. This benchmark dataset contains text and speech samples from social media in Tamil, Malayalam, and Telugu, labeled into six classes: Non-Hate, Gender Hate, Political Hate, Religious Hate, Religious Defamation, and Personal Defamation. The paper highlights the inherent class imbalance within this dataset.
Key Insights and Ideas
- Multimodality is Crucial: The paper strongly suggests that combining text and speech is essential for effective hate speech detection, especially in languages where one modality might be less informative or more prone to ambiguity.
- Late Fusion Outperforms Early Fusion: While early fusion (concatenating features before classification) was explored, late fusion proved to be more effective in terms of generalization. This indicates that allowing independent optimization of text and speech models before combining their predictions is a more robust strategy.
- Class Imbalance is a Major Hurdle: The significant class imbalance in the dataset poses a substantial challenge, requiring specific techniques like class weighting and data augmentation to achieve reasonable performance.
- Telugu's Unique Challenges: Telugu exhibits the weakest performance across all models, suggesting that its specific linguistic characteristics (phonetic and syntactic diversity) and data sparsity present unique difficulties for both text and speech processing models, even advanced ones like Wav2Vec 2.0.
Key Results
- Late Fusion Generalization: Late fusion, particularly with class weighting, showed improved generalization compared to early fusion. For instance, test F1-scores for Tamil and Malayalam reached 0.71 and 0.75 respectively with late fusion and class weighting (Table 2).
- Data Augmentation Benefits: Data augmentation improved robustness, especially for Tamil, where the late fusion F1-score reached 0.70 after augmentation (Table 3).
- Early Fusion Overfitting: Early fusion exhibited significant overfitting, with a large drop in performance from the training set (0.79-0.83) to the test set (0.40-0.51) (Table 1).
- Telugu's Underperformance: Telugu's performance remained weak, with late fusion achieving a low F1-score of 0.17 in some instances, highlighting its underperformance across all modalities.
- Class-Specific Performance: The classification report (Table 4) shows varying performance across hate speech categories. While classes like Religious (R) and Political (P) hate had good recall and F1-scores, Gender Hate (G) suffered from poor recall (0.30), indicating difficulty in identifying these instances.
Limitations and Future Work
- Class Disparity: The paper acknowledges that class disparity remains a significant problem, particularly for political and religious hate categories.
- Speech Processing Difficulties: The lack of pre-trained models for Dravidian languages, background noise, and accent fluctuations make speech processing challenging.
- Modality Imbalance: The text-based model generally performed better than the speech-based model, leading to an imbalance in modality contributions that late fusion couldn't fully overcome.
- Generalization Challenges: Changes in hate speech patterns and a lack of discourse-level knowledge hinder generalization to real-world contexts.
- Future Work: The authors suggest focusing on reducing overfitting with regularization techniques, evaluating more advanced transformer models, and improving interpretability for better linguistic adaptation. They also plan to make their implementation publicly available.
Summary updated